public/openhab-addons
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[dynamodb] Dynamodb refactor (#9937)

Browse Source 
* [dynamodb] Update to SDKv2 Enhanced Client

In addition, introduce new more simple table layout, having only one
table for all items and with more efficient data encoding (saves some read capacity).

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Time To Live (TTL) support with new table schema

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Support QuantityType

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] suppress null warnings in tests

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Optimized query performance

Similar to https://github.com/openhab/openhab-addons/pull/8938,
avoid calling Item.getUnit() repeatedly when querying data.

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Support for Group items

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Update copyright to 2021

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Removing TODO comments and add javadoc

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] javadoc

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Readability improved in TableCreatingPutItem

Also documenting the full retry logic.

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] verify fixes

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Remove slf4j from explicit dependencies

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Remove jackson from pom.xml, add as feature dep

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] bnd.importpackage tuned

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] abort query() immediately if not configured to avoid NPE

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] less chatty diagnostics

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] xml formatting

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] corrected logger class

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] null checks

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] netty client configured

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] bnd not to filter out importpackage org.slf4j.impl

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] cfg bundle group id

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Remove usage of org.apache.commons

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Remove extra prints from test

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Reducing @SupressWarnings with generics

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] README extra space removed

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] spotless

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Removed unnecessary logging

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] encapsulation

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] removed unnecessary NonNullByDefault({}) ctr-injected field

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] null annotations

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] less verbose logging in tests

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Prefer Collections.emptyList over List.of()

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] less verbose call

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Visitor to return values (simplifies the code)

Less warnings suppressed

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] comments for remaining warning supressions

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] README tuning, typo fixing

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Using less verbose syntax

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] simplified logging on errors

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Code review comments

Avoiding null checker while having more compact code

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Null safety

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] configuration label and description formatting

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] xml indentation with tabs

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] @Nullable 1-line annotation with class fields

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] No need to override credentials per request

Client has the credentials set on build time

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] set API timeouts no matter what

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] adding exception message

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] static logger

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] dependency

- comments clarifying the logic of properties
- adding netty to dep.noembedding to ensure it is not compiled in

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] ensure correct jackson and netty versions using dependencyMgt

Specifically for development and testing

See 051c764789
for further discussion why this is needed.

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] avoid google collections

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] jackson-dataformat-cbor not jackson-cbor

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] also restrict netty-transport-native-epoll linux-x86_64 version

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] refering dynamodb.cfg similar to other bundles

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] bnd.importpackage to excl. reactivestreams and typesafe.netty

These are compiled-in dependencies, and thus we do not want to have them in
OSGi Import-Package.

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* Update bundles/org.openhab.persistence.dynamodb/src/main/resources/OH-INF/config/config.xml

Co-authored-by: Fabian Wolter <github@fabian-wolter.de>
Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* Update bundles/org.openhab.persistence.dynamodb/src/main/resources/OH-INF/config/config.xml

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

Co-authored-by: Fabian Wolter <github@fabian-wolter.de>

* [dynamodb] remove netty-codec-http2 as it is included in tp-netty

See https://github.com/openhab/openhab-core/pull/2257/

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] removed duplicate in bnd.importpackage

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] slf4j-api marked as provided to remove dep errors in runtime

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

Co-authored-by: Fabian Wolter <github@fabian-wolter.de>
This commit is contained in:
Sami Salonen
2021-04-10 23:13:38 +03:00
committed by GitHub
parent 08602c04b4
commit b675160486
58 changed files with 4407 additions and 1592 deletions
Download Patch File Download Diff File Expand all files Collapse all files

120
bundles/org.openhab.persistence.dynamodb/README.md
View File

@@ -15,22 +15,10 @@ This service is provided "AS IS", and the user takes full responsibility of any
## Table of Contents
<!-- Using MarkdownTOC plugin for Sublime Text to update the table of contents (TOC) -->
<!-- MarkdownTOC depth=3 autolink=true bracket=round -->
{::options toc_levels="2..4"/}
- [Prerequisites](#prerequisites)
- [Setting Up an Amazon Account](#setting-up-an-amazon-account)
- [Configuration](#configuration)
- [Basic configuration](#basic-configuration)
- [Configuration Using Credentials File](#configuration-using-credentials-file)
- [Advanced Configuration](#advanced-configuration)
- [Details](#details)
- [Tables Creation](#tables-creation)
- [Caveats](#caveats)
- [Developer Notes](#developer-notes)
- [Updating Amazon SDK](#updating-amazon-sdk)
<!-- /MarkdownTOC -->
- TOC
{:toc}
## Prerequisites
@@ -55,9 +43,53 @@ Please also note possible [Free Tier](https://aws.amazon.com/free/) benefits.
## Configuration
This service can be configured in the file `services/dynamodb.cfg`.
This service can be configured using the MainUI or using persistence configuration file `services/dynamodb.cfg`.
### Basic configuration
In order to configure the persistence service, you need to configure two things:
1. Table schema revision to use
2. AWS credentials to access DynamoDB
### Table schema
The DynamoDB persistence addon provides two different table schemas: "new" and "legacy".
As the name implies, "legacy" is offered for backwards-compatibility purpose for old users who like to access the data that is already stored in DynamoDB.
All users are advised to transition to "new" table schema, which is more optimized.
At this moment there is no supported way to migrate data from old format to new.
#### New table schema
Configure the addon to use new schema by setting `table` parameter (name of the table).
Only one table will be created for all data. The table will have the following fields
| Attribute | Type | Data type | Description |
| --------- | ------ | --------- | --------------------------------------------- |
| `i` | String | Yes | Item name |
| `t` | Number | Yes | Timestamp in milliepoch |
| `s` | String | Yes | State of the item, stored as DynamoDB string. |
| `n` | Number | Yes | State of the item, stored as DynamoDB number. |
| `exp` | Number | Yes | Expiry date for item, in epoch seconds |
Other notes
- `i` and `t` forms the composite primary key (partition key, sort key) for the table
- Only one of `s` or `n` attributes are specified, not both. Most items are converted to number type for most compact representation.
- Compared to legacy format, data overhead is minimizing by using short attribute names, number timestamps and having only single table.
- `exp` attribute is used with DynamoDB Time To Live (TTL) feature to automatically delete old data
#### Legacy schema
Configure the addon to use legacy schema by setting `tablePrefix` parameter.
- When an item is persisted via this service, a table is created (if necessary).
- The service will create at most two tables for different item types.
- The tables will be named `<tablePrefix><item-type>`, where the `<item-type>` is either `bigdecimal` (numeric items) or `string` (string and complex items).
- Each table will have three columns: `itemname` (item name), `timeutc` (in ISO 8601 format with millisecond accuracy), and `itemstate` (either a number or string representing item state).
### Credentials Configuration Using Access Key and Secret Key
| Property | Default | Required | Description |
| --------- | ------- | :------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -65,7 +97,7 @@ This service can be configured in the file `services/dynamodb.cfg`.
| secretKey | | Yes | secret key as shown in [Setting up Amazon account](#setting-up-an-amazon-account). |
| region | | Yes | AWS region ID as described in [Setting up Amazon account](#setting-up-an-amazon-account). The region needs to match the region that was used to create the user. |
### Configuration Using Credentials File
### Credentials Configuration Using Credentials File
Alternatively, instead of specifying `accessKey` and `secretKey`, one can configure a configuration profile file.
@@ -95,47 +127,27 @@ aws_secret_access_key=testSecretKey
In addition to the configuration properties above, the following are also available:
| Property | Default | Required | Description |
| -------------------------- | ---------- | :------: | -------------------------------------------------------------------------------------------------- |
| readCapacityUnits | 1 | No | read capacity for the created tables |
| writeCapacityUnits | 1 | No | write capacity for the created tables |
| tablePrefix | `openhab-` | No | table prefix used in the name of created tables |
| bufferCommitIntervalMillis | 1000 | No | Interval to commit (write) buffered data. In milliseconds. |
| bufferSize | 1000 | No | Internal buffer size in datapoints which is used to batch writes to DynamoDB every `bufferCommitIntervalMillis`. |
Typically you should not need to modify parameters related to buffering.
| Property | Default | Required | Description |
| ------------------ | ------- | :------: | ----------------------------------------------------------- |
| expireDays | (null) | No | Expire time for data in days (relative to stored timestamp) |
| readCapacityUnits | 1 | No | read capacity for the created tables |
| writeCapacityUnits | 1 | No | write capacity for the created tables |
Refer to Amazon documentation on [provisioned throughput](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html) for details on read/write capacity.
DynamoDB Time to Live (TTL) setting is configured using `expireDays`.
All item- and event-related configuration is done in the file `persistence/dynamodb.persist`.
## Details
### Tables Creation
When an item is persisted via this service, a table is created (if necessary).
Currently, the service will create at most two tables for different item types.
The tables will be named `<tablePrefix><item-type>`, where the `<item-type>` is either `bigdecimal` (numeric items) or `string` (string and complex items).
Each table will have three columns: `itemname` (item name), `timeutc` (in ISO 8601 format with millisecond accuracy), and `itemstate` (either a number or string representing item state).
## Buffering
By default, the service is asynchronous which means that data is not written immediately to DynamoDB but instead buffered in-memory.
The size of the buffer, in terms of datapoints, can be configured with `bufferSize`.
Every `bufferCommitIntervalMillis` the whole buffer of data is flushed to DynamoDB.
It is recommended to have the buffering enabled since the synchronous behaviour (writing data immediately) might have adverse impact to the whole system when there is many items persisted at the same time.
The buffering can be disabled by setting `bufferSize` to zero.
The defaults should be suitable in many use cases.
### Caveats
When the tables are created, the read/write capacity is configured according to configuration.
However, the service does not modify the capacity of existing tables.
As a workaround, you can modify the read/write capacity of existing tables using the [Amazon console](https://aws.amazon.com/console/).
Similar caveat applies for DynamoDB Time to Live (TTL) setting `expireDays`.
## Developer Notes
### Updating Amazon SDK
@@ -143,20 +155,15 @@ As a workaround, you can modify the read/write capacity of existing tables using
1. Clean `lib/*`
2. Update SDK version in `scripts/fetch_sdk_pom.xml`. You can use the [maven online repository browser](https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dynamodb) to find the latest version available online.
3. `scripts/fetch_sdk.sh`
4. Copy `scripts/target/site/dependencies.html` and `scripts/target/dependency/*.jar` to `lib/`
5. Generate `build.properties` entries
`ls lib/*.jar | python -c "import sys; print(' ' + ',\\\\\\n '.join(map(str.strip, sys.stdin.readlines())))"`
6. Generate `META-INF/MANIFEST.MF` `Bundle-ClassPath` entries
`ls lib/*.jar | python -c "import sys; print(' ' + ',\\n '.join(map(str.strip, sys.stdin.readlines())))"`
7. Generate `.classpath` entries
`ls lib/*.jar | python -c "import sys;pre='<classpathentry exported=\"true\" kind=\"lib\" path=\"';post='\"/>'; print('\\t' + pre + (post + '\\n\\t' + pre).join(map(str.strip, sys.stdin.readlines())) + post)"`
4. Copy printed dependencies to `pom.xml`
After these changes, it's good practice to run integration tests (against live AWS DynamoDB) in `org.openhab.persistence.dynamodb.test` bundle.
See README.md in the test bundle for more information how to execute the tests.
### Running integration tests
To run integration tests, one needs to provide AWS credentials.
When running integration tests, local temporary DynamoDB server is used, emulating the real AWS DynamoDB API.
One can configure AWS credentials to run the test against real AWS DynamoDB for most realistic tests.
Eclipse instructions
@@ -168,7 +175,4 @@ Eclipse instructions
-DDYNAMODBTEST_REGION=REGION-ID
-DDYNAMODBTEST_ACCESS=ACCESS-KEY
-DDYNAMODBTEST_SECRET=SECRET
````
The tests will create tables with prefix `dynamodb-integration-tests-`.
Note that when tests are begun, all data is removed from that table!
````