The primary function of this tooling is to query a set of tags for a specified date range, compress and process those values as required, and move it into a format which can be easily imported into VTScada.
## Setup
### TagMap file
The TagMap file is a CSV file of which tags to query on the existing system. They are queried with the following paramaters.
* Interval - time in seconds to query the data point
* Deadband - values within this deadband before scaling will be elimianted
* Scale Factor - scaling factor to apply to existing data before compression
* Precision - the final result value will be rounded to this many decimal places
For all values, values will either be working from a set of all data (for example, ClearSCADA), or querieid at the required interval (for example, VTScada REST). For the former, values within the interval will be deleted after compression.
For boolean values, they will be **sampled** at the interval specified and compressed by eliminating values which stay the same. To ensure the precision of transitions wanted is kept, a high precision such as 1 second sampling is recommended here
For integer/analog values, they will be **averaged** at the interval specified, and compressed by eliminating values within the deadband, then scaled, then rounded to the required precision. A lower precision of sampling is recommended here, perhaps 5 seconds for high-value points, and 30-60 seconds for low value points.
### Setup TOML
This file contains configuration on how to connect to the required existing interfaces and locations of input and the output data.
### Python
This system requires Python installed and run on the system which can target existing interfaces. Several libraries will need to be installed by pip.
## VTScada REST
This is a method of moving VTScada data into VTScada data. Scenarios where this could be useful:
* Moving tag data from Analog Status / Digital Status to the modern IO tags
* Tag data sizes which have gotten out of hand
* A site has been recommissioned with a new set of tags and data needs to be imported
## ClearSCADA - Raw Historic Files
In places where targetting a live ClearSCADA system with SQL queries is challenging, ClearSCADA uses a file-based Historian and provides a utility which converts these HRD files into CSV data.
Each directory contains a Historic XXXXXX directory where XXXXX is the Unique ID of the datapoint padded with a file WKYYYYYY where YYYYYY is the number of weeks since January 1, 1601 (yes, really).
These tools will conver the user start time and end time in a way that will only process the found and required HRD files at a time. This can *greatly* expand the amount of data in the system, it is strongly recommended to have a lot of free space left during queries.
### Setup ClearSCADA Config
Key Point: In the tags list file, the Source Name field is the unique identifier for the tag name to query. In VTScada this can be something like ```temp\old_value1```. In ClearSCADA, it will be the unique point ID, ex. ```005152```. The leading zeroes can be left out as the script will pad them in front of the integer to determine the correct path.
Prerequisite: Install the Microsoft ODBC Driver, and assign the correct name of the ODBC driver (visible in the ODBC Data Sources Windows application) to the code, if the default one is not detected.
Data can be queried from an AVEVA Historian through an ODBC connection. The Historian provides an interface to SQL, InSQL, which can be used to create data queries.
Wonderware Historian queries are extermely slow, but in practice I have found splitting it up into time periods such as 6 months will scale to running at least dozens of queries at once.
