# VTScada-HistoricalTools

This is a set of tooling to perform useful operations of queryign and moving Historical data into ClearSCADA

Currently supports querying data from the following sources, and their requirements
* VTScada - REST
* ClearSCADA - Raw Historic Files
* AVEVA (Wonderware) Historian - InSQL {Coming Soon}

The primary function of this tooling is to query a set of tags for a specified date range, compress and process those values as required, and move it into a format which can be easily imported into VTScada.

## Setup

### TagMap file

The TagMap file is a CSV file of which tags to query on the existing system. They are queried with the following paramaters.
* Interval - time in seconds to query the data point
* Deadband - values within this deadband before scaling will be elimianted
* Scale Factor - scaling factor to apply to existing data before compression
* Precision - the final result value will be rounded to this many decimal places

For all values, values will either be working from a set of all data (for example, ClearSCADA), or querieid at the required interval (for example, VTScada REST). For the former, values within the interval will be deleted after compression.

For boolean values, they will be **sampled** at the interval specified and compressed by eliminating values which stay the same. To ensure the precision of transitions wanted is kept, a high precision such as 1 second sampling is recommended here

For integer/analog values, they will be **averaged** at the interval specified, and compressed by eliminating values within the deadband, then scaled, then rounded to the required precision. A lower precision of sampling is recommended here, perhaps 5 seconds for high-value points, and 30-60 seconds for low value points.

### Setup TOML

This file contains configuration on how to connect to the required existing interfaces and locations of input and the output data.

### Python

This system requires Python installed and run on the system which can target existing interfaces. Several libraries will need to be installed by pip.

## VTScada REST

This is a method of moving VTScada data into VTScada data. Scenarios where this could be useful:
* Moving tag data from Analog Status / Digital Status to the modern IO tags
* Tag data sizes which have gotten out of hand
* A site has been recommissioned with a new set of tags and data needs to be imported

## ClearSCADA - Raw Historic Files

In places where targetting a live ClearSCADA system with SQL queries is challenging, ClearSCADA uses a file-based Historian and provides a utility which converts these HRD files into CSV data. 

For each week of each data point, a separate CSV file of data is created from the 

Files are generally stored:
```C:\ProgramData\Schneider Electric\ClearSCADA\Database\HisFiles```

Each directory contains a Historic XXXXXX directory where XXXXX is the Unique ID of the datapoint padded with a file WKYYYYYY where YYYYYY is the number of weeks since January 1, 1601 (yes, really). 

These tools will conver the user start time and end time in a way that will only process the found and required HRD files at a time. This can *greatly* expand the amount of data in the system, it is strongly recommended to have a lot of free space left during queries.

### Setup ClearSCADA Config 

Key Point: In the tags list file, the Source Name field is the unique identifier for the tag name to query. In VTScada this can be something like ```temp\old_value1```. In ClearSCADA, it will be the unique point ID, ex. ```005152```. The leading zeroes can be left out as the script will pad them in front of the integer to determine the correct path.