# Amazon DynamoDB Persistence This service allows you to persist state updates using the [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) database. Query functionality is also fully supported. Features: * Writing/reading information to relational database systems * Configurable database table names * Automatic table creation ## Disclaimer This service is provided "AS IS", and the user takes full responsibility of any charges or damage to Amazon data. ## Table of Contents - [Prerequisites](#prerequisites) - [Setting Up an Amazon Account](#setting-up-an-amazon-account) - [Configuration](#configuration) - [Basic configuration](#basic-configuration) - [Configuration Using Credentials File](#configuration-using-credentials-file) - [Advanced Configuration](#advanced-configuration) - [Details](#details) - [Tables Creation](#tables-creation) - [Caveats](#caveats) - [Developer Notes](#developer-notes) - [Updating Amazon SDK](#updating-amazon-sdk) ## Prerequisites You must first set up an Amazon account as described below. Users are recommended to familiarize themselves with AWS pricing before using this service. Please note that there might be charges from Amazon when using this service to query/store data to DynamoDB. See [Amazon DynamoDB pricing pages](https://aws.amazon.com/dynamodb/pricing/) for more details. Please also note possible [Free Tier](https://aws.amazon.com/free/) benefits. ### Setting Up an Amazon Account * [Sign up](https://aws.amazon.com/) for Amazon AWS. * Select the AWS region in the [AWS console](https://console.aws.amazon.com/) using [these instructions](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/getting-started.html#select-region). Note the region identifier in the URL (e.g. `https://eu-west-1.console.aws.amazon.com/console/home?region=eu-west-1` means that region id is `eu-west-1`). * **Create user for openHAB with IAM** * Open Services -> IAM -> Users -> Create new Users. Enter `openhab` to _User names_, keep _Generate an access key for each user_ checked, and finally click _Create_. * _Show User Security Credentials_ and record the keys displayed * **Configure user policy to have access for dynamodb** * Open Services -> IAM -> Policies * Check _AmazonDynamoDBFullAccess_ and click _Policy actions_ -> _Attach_ * Check the user created in step 2 and click _Attach policy_ ## Configuration This service can be configured in the file `services/dynamodb.cfg`. ### Basic configuration | Property | Default | Required | Description | | --------- | ------- | :------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | accessKey | | Yes | access key as shown in [Setting up Amazon account](#setting-up-an-amazon-account). | | secretKey | | Yes | secret key as shown in [Setting up Amazon account](#setting-up-an-amazon-account). | | region | | Yes | AWS region ID as described in [Setting up Amazon account](#setting-up-an-amazon-account). The region needs to match the region that was used to create the user. | ### Configuration Using Credentials File Alternatively, instead of specifying `accessKey` and `secretKey`, one can configure a configuration profile file. | Property | Default | Required | Description | | ------------------ | ------- | :------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | profilesConfigFile | | Yes | path to the credentials file. For example, `/etc/openhab2/aws_creds`. Please note that the user that runs openHAB must have approriate read rights to the credential file. For more details on the Amazon credential file format, see [Amazon documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). | | profile | | Yes | name of the profile to use | | region | | Yes | AWS region ID as described in Step 2 in [Setting up Amazon account](#setting-up-an-amazon-account). The region needs to match the region that was used to create the user. | Example of service configuration file (`services/dynamodb.cfg`): ```ini profilesConfigFile=/etc/openhab2/aws_creds profile=fooprofile region=eu-west-1 ``` Example of credentials file (`/etc/openhab2/aws_creds`): ````ini [fooprofile] aws_access_key_id=testAccessKey aws_secret_access_key=testSecretKey ```` ### Advanced Configuration In addition to the configuration properties above, the following are also available: | Property | Default | Required | Description | | -------------------------- | ---------- | :------: | -------------------------------------------------------------------------------------------------- | | readCapacityUnits | 1 | No | read capacity for the created tables | | writeCapacityUnits | 1 | No | write capacity for the created tables | | tablePrefix | `openhab-` | No | table prefix used in the name of created tables | | bufferCommitIntervalMillis | 1000 | No | Interval to commit (write) buffered data. In milliseconds. | | bufferSize | 1000 | No | Internal buffer size in datapoints which is used to batch writes to DynamoDB every `bufferCommitIntervalMillis`. | Typically you should not need to modify parameters related to buffering. Refer to Amazon documentation on [provisioned throughput](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html) for details on read/write capacity. All item- and event-related configuration is done in the file `persistence/dynamodb.persist`. ## Details ### Tables Creation When an item is persisted via this service, a table is created (if necessary). Currently, the service will create at most two tables for different item types. The tables will be named ``, where the `` is either `bigdecimal` (numeric items) or `string` (string and complex items). Each table will have three columns: `itemname` (item name), `timeutc` (in ISO 8601 format with millisecond accuracy), and `itemstate` (either a number or string representing item state). ## Buffering By default, the service is asynchronous which means that data is not written immediately to DynamoDB but instead buffered in-memory. The size of the buffer, in terms of datapoints, can be configured with `bufferSize`. Every `bufferCommitIntervalMillis` the whole buffer of data is flushed to DynamoDB. It is recommended to have the buffering enabled since the synchronous behaviour (writing data immediately) might have adverse impact to the whole system when there is many items persisted at the same time. The buffering can be disabled by setting `bufferSize` to zero. The defaults should be suitable in many use cases. ### Caveats When the tables are created, the read/write capacity is configured according to configuration. However, the service does not modify the capacity of existing tables. As a workaround, you can modify the read/write capacity of existing tables using the [Amazon console](https://aws.amazon.com/console/). ## Developer Notes ### Updating Amazon SDK 1. Clean `lib/*` 2. Update SDK version in `scripts/fetch_sdk_pom.xml`. You can use the [maven online repository browser](https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dynamodb) to find the latest version available online. 3. `scripts/fetch_sdk.sh` 4. Copy `scripts/target/site/dependencies.html` and `scripts/target/dependency/*.jar` to `lib/` 5. Generate `build.properties` entries `ls lib/*.jar | python -c "import sys; print(' ' + ',\\\\\\n '.join(map(str.strip, sys.stdin.readlines())))"` 6. Generate `META-INF/MANIFEST.MF` `Bundle-ClassPath` entries `ls lib/*.jar | python -c "import sys; print(' ' + ',\\n '.join(map(str.strip, sys.stdin.readlines())))"` 7. Generate `.classpath` entries `ls lib/*.jar | python -c "import sys;pre=''; print('\\t' + pre + (post + '\\n\\t' + pre).join(map(str.strip, sys.stdin.readlines())) + post)"` After these changes, it's good practice to run integration tests (against live AWS DynamoDB) in `org.openhab.persistence.dynamodb.test` bundle. See README.md in the test bundle for more information how to execute the tests. ### Running integration tests To run integration tests, one needs to provide AWS credentials. Eclipse instructions 1. Run all tests (in package org.openhab.persistence.dynamodb.internal) as JUnit Tests 2. Configure the run configuration, and open Arguments sheet 3. In VM arguments, provide the credentials for AWS ```` -DDYNAMODBTEST_REGION=REGION-ID -DDYNAMODBTEST_ACCESS=ACCESS-KEY -DDYNAMODBTEST_SECRET=SECRET ```` The tests will create tables with prefix `dynamodb-integration-tests-`. Note that when tests are begun, all data is removed from that table!