Updating with a few bug fixes and cleanup for AVEVA processing, updating Python deprecated functions.

This commit is contained in:
Michael Van Ryn 2024-08-15 11:15:54 -06:00
parent 4c8372d7b1
commit f439d171b5
5 changed files with 167 additions and 34 deletions

View File

@ -53,13 +53,14 @@ Each directory contains a Historic XXXXXX directory where XXXXX is the Unique ID
These tools will conver the user start time and end time in a way that will only process the found and required HRD files at a time. This can *greatly* expand the amount of data in the system, it is strongly recommended to have a lot of free space left during queries. These tools will conver the user start time and end time in a way that will only process the found and required HRD files at a time. This can *greatly* expand the amount of data in the system, it is strongly recommended to have a lot of free space left during queries.
### Setup ClearSCADA Config ### Setup ClearSCADA Config
Key Point: In the tags list file, the Source Name field is the unique identifier for the tag name to query. In VTScada this can be something like ```temp\old_value1```. In ClearSCADA, it will be the unique point ID, ex. ```005152```. The leading zeroes can be left out as the script will pad them in front of the integer to determine the correct path. Key Point: In the tags list file, the Source Name field is the unique identifier for the tag name to query. In VTScada this can be something like ```temp\old_value1```. In ClearSCADA, it will be the unique point ID, ex. ```005152```. The leading zeroes can be left out as the script will pad them in front of the integer to determine the correct path.
## AVEVA (Wonderware) Historian - InSQL ## AVEVA (Wonderware) Historian - InSQL
Prerequisite: Install the Microsoft ODBC Driver, and assign the correct name to the driver in the *setup.toml* file. Prerequisite: Install the Microsoft ODBC Driver, and assign the correct name of the ODBC driver (visible in the ODBC Data Sources Windows application) to the code, if the default one is not detected.
Data can be queried from an AVEVA Historian through an ODBC connection. The Historian provides an interface to SQL, InSQL, which can be used to create data queries. Data can be queried from an AVEVA Historian through an ODBC connection. The Historian provides an interface to SQL, InSQL, which can be used to create data queries.
Wonderware Historian queries are extermely slow, but in practice I have found splitting it up into time periods such as 6 months will scale to running at least dozens of queries at once.

17
input/tags-example.csv Normal file
View File

@ -0,0 +1,17 @@
Source Name,Destination Name,Data Type,Scale Factor,Interval (s),Precision,Deadband
11959,temp\new_value1,real,1,20,2,0.01
5153,temp\new_value2,integer,1,100,0,0
5154,temp\new_value3,real,1,30,0,1
5155,temp\new_value4,boolean,1,9,0,0
5,temp\new_value5,real,1,20,2,0
6,temp\new_value6,integer,10,100,0,0
227,temp\new_value7,real,1.5,100,2,0
8,temp\new_value8,boolean,1,9,0,0
9,temp\new_value9,real,1,10,2,0
10,temp\new_value10,integer,1,10,0,0
011,temp\new_value11,real,1,10,4,0
12,temp\new_value12,boolean,1,10,0,0
1113,temp\new_value13,real,1,12,4,0
14,temp\new_value14,integer,1,13,0,0
1665,temp\new_value15,real,1,14,4,0
16,temp\new_value16,boolean,1,15,0,0
1 Source Name Destination Name Data Type Scale Factor Interval (s) Precision Deadband
2 11959 temp\new_value1 real 1 20 2 0.01
3 5153 temp\new_value2 integer 1 100 0 0
4 5154 temp\new_value3 real 1 30 0 1
5 5155 temp\new_value4 boolean 1 9 0 0
6 5 temp\new_value5 real 1 20 2 0
7 6 temp\new_value6 integer 10 100 0 0
8 227 temp\new_value7 real 1.5 100 2 0
9 8 temp\new_value8 boolean 1 9 0 0
10 9 temp\new_value9 real 1 10 2 0
11 10 temp\new_value10 integer 1 10 0 0
12 011 temp\new_value11 real 1 10 4 0
13 12 temp\new_value12 boolean 1 10 0 0
14 1113 temp\new_value13 real 1 12 4 0
15 14 temp\new_value14 integer 1 13 0 0
16 1665 temp\new_value15 real 1 14 4 0
17 16 temp\new_value16 boolean 1 15 0 0

View File

@ -1,17 +1,124 @@
Source Name,Destination Name,Data Type,Scale Factor,Interval (s),Precision,Deadband Source Name,Destination Name,Data Type,Scale Factor,Interval (s),Precision,Deadband
11959,temp\new_value1,real,1,20,2,0.01 AHU6_Z_SP_3000571.Value,_\BACNET_HVAC\AHU6\RM_TEMP_SP\CMD,integer,1,100,0,0
5153,temp\new_value2,integer,1,100,0,0 Airport_Boost_Pump_002.Runtime,_\DRS\Airport_BS\P102\RUN_TIME\HOURS,real,1,60,1,0.1
5154,temp\new_value3,real,1,30,0,1 Airport_Boost_Pump_004.Runtime,_\DRS\Airport_BS\P104\RUN_TIME\HOURS,real,1,60,1,0.1
5155,temp\new_value4,boolean,1,9,0,0 Airport_Boost_Pump_001.Runtime,_\DRS\Airport_BS\P101\RUN_TIME\HOURS,real,1,60,1,0.1
5,temp\new_value5,real,1,20,2,0 Airport_Boost_Pump_003.Runtime,_\DRS\Airport_BS\P103\RUN_TIME\HOURS,real,1,60,1,0.1
6,temp\new_value6,integer,10,100,0,0 AHU101_DA_TEMP_3000340.Value,_\BACNET_HVAC\AHU101\SA_TEMP\SCL,real,1,60,1,0.1
227,temp\new_value7,real,1.5,100,2,0 AHU6_DA_TEMP_3000446.Value,_\BACNET_HVAC\AHU6\SA_TEMP\SCL,real,1,60,1,0.1
8,temp\new_value8,boolean,1,9,0,0 Airport_Boost_Pump_001.Current_Demand,_\DRS\Airport_BS\CT_101\SCL,real,1,60,1,0.1
9,temp\new_value9,real,1,10,2,0 Airport_Boost_Pump_002.Current_Demand,_\DRS\Airport_BS\CT_102\SCL,real,1,60,1,0.1
10,temp\new_value10,integer,1,10,0,0 Airport_Boost_Pump_003.Current_Demand,_\DRS\Airport_BS\CT_103\SCL,real,1,60,1,0.1
011,temp\new_value11,real,1,10,4,0 Airport_Boost_Pump_004.Current_Demand,_\DRS\Airport_BS\CT_104\SCL,real,1,60,1,0.1
12,temp\new_value12,boolean,1,10,0,0 Airport_System.Discharge_Flow,_\DRS\Airport_BS\FIT_101\SCL,real,1,60,1,0.1
1113,temp\new_value13,real,1,12,4,0 Airport_System.Total_Flow,_\DRS\Airport_BS\FIT_101\TOTALIZE,real,1,60,1,0.1
14,temp\new_value14,integer,1,13,0,0 Airport_Booster.P101_SPEED,_\DRS\Airport_BS\P101\YI,real,1,60,1,0.1
1665,temp\new_value15,real,1,14,4,0 Airport_Booster.P102_SPEED,_\DRS\Airport_BS\P102\YI,real,1,60,1,0.1
16,temp\new_value16,boolean,1,15,0,0 Airport_Booster.P103_SPEED,_\DRS\Airport_BS\P103\YI,real,1,60,1,0.1
Airport_Booster.P104_SPEED,_\DRS\Airport_BS\P104\YI,real,1,60,1,0.1
Airport_System.Discharge_Pressure,_\DRS\Airport_BS\PIT_102\SCL,real,1,60,1,0.1
Southlands_Lift.FIT_102,_\DRS\Southlands_LS\FIT_102\SCL,real,1,60,1,0.1
Southlands_Lift.LT_127A,_\DRS\Southlands_LS\LT_127A\SCL,real,1,60,1,0.1
Southridge_Res.LIT_219_mm,_\DRS\Southridge_Reservoir\LIT_219\SCL,real,1,60,1,0.1
Southview_Res.Chlorine_Residual,_\DRS\Southview_Reservoir\AIT_101\SCL,real,1,60,1,0.1
Southview_System.Chlorine_Residual,_\DRS\Southview_Reservoir\AIT_101\SCL,real,1,60,1,0.1
Southview_Res.Pump_1_Speed_Hz,_\DRS\Southview_Reservoir\BP_101\YI,real,1,60,1,0.1
Southview_Res.Pump_2_Speed_Hz,_\DRS\Southview_Reservoir\BP_102\YI,real,1,60,1,0.1
Southview_Res.Pump_3_Speed_Hz,_\DRS\Southview_Reservoir\BP_103\YI,real,1,60,1,0.1
Southview_Res.CT_101,_\DRS\Southview_Reservoir\CT_101\SCL,real,1,60,1,0.1
Southview_Res.CT_102,_\DRS\Southview_Reservoir\CT_102\SCL,real,1,60,1,0.1
Southview_Res.CT_103,_\DRS\Southview_Reservoir\CT_103\SCL,real,1,60,1,0.1
Southview_Res.FIT_101,_\DRS\Southview_Reservoir\FIT_101\SCL,real,1,60,1,0.1
Southview_System.Inlet_Flow,_\DRS\Southview_Reservoir\FIT_101\SCL,real,1,60,1,0.1
Southview_Res.FIT_102,_\DRS\Southview_Reservoir\FIT_102\SCL,real,1,60,1,0.1
Southview_System.Discharge_Flow,_\DRS\Southview_Reservoir\FIT_102\SCL,real,1,60,1,0.1
Southview_Res.LT_101,_\DRS\Southview_Reservoir\LIT_101\SCL,real,1,60,1,0.1
Southview_System.Level,_\DRS\Southview_Reservoir\LIT_101\SCL,real,1,60,1,0.1
Southview_Res.LT_101_mm,_\DRS\Southview_Reservoir\LIT_101\SCL2,real,1,60,1,0.1
Southview_System.Level_m,_\DRS\Southview_Reservoir\LIT_101\SCL2,real,1,60,1,0.1
Southview_Res.PT_101,_\DRS\Southview_Reservoir\PIT_101\SCL,real,1,60,1,0.1
Southview_System.Inlet_Pressure,_\DRS\Southview_Reservoir\PIT_101\SCL,real,1,60,1,0.1
Southview_Res.PT_102,_\DRS\Southview_Reservoir\PIT_102\SCL,real,1,60,1,0.1
Southview_System.Discharge_Pressure,_\DRS\Southview_Reservoir\PIT_102\SCL,real,1,60,1,0.1
AIT_DU_8638.SCL,_\WTP\ChlorScrubber\AIT_DU_8638\SCL,real,1,60,1,0.1
AIT_DU_8639.SCL,_\WTP\ChlorScrubber\AIT_DU_8639\SCL,real,1,60,1,0.1
AIT_DU_8640.SCL,_\WTP\ChlorScrubber\AIT_DU_8640\SCL,real,1,60,1,0.1
AIT_DU_8641.SCL,_\WTP\ChlorScrubber\AIT_DU_8641\SCL,real,1,60,1,0.1
AIT_DU_8642.SCL,_\WTP\ChlorScrubber\AIT_DU_8642\SCL,real,1,60,1,0.1
AIT_DU_8643.SCL,_\WTP\ChlorScrubber\AIT_DU_8643\SCL,real,1,60,1,0.1
AIT_DU_8644.SCL,_\WTP\ChlorScrubber\AIT_DU_8644\SCL,real,1,60,1,0.1
AIT_DU_8645.SCL,_\WTP\ChlorScrubber\AIT_DU_8645\SCL,real,1,60,1,0.1
AIT8809.SCL,_\WTP\Dechlor\Tank_12\AIT8809\SCL,real,1,60,1,0.1
AIT8811.SCL,_\WTP\Dechlor\Tank_12\AIT8811\SCL,real,1,60,1,0.1
AIT8803.SCL,_\WTP\Dechlor\Tank_3\AIT8803\SCL,real,1,60,1,0.1
AIT8805.SCL,_\WTP\Dechlor\Tank_3\AIT8805\SCL,real,1,60,1,0.1
SCU_1.Effluent_Turbidity,_\WTP\SCU1\EFF_NTU\SCL,real,1,60,3,0.001
SCU_1.Effluent_pH,_\WTP\SCU1\EFF_PH\SCL,real,1,60,2,0.01
Act_Carb_Tank_001.Tank_Level_L,_\WTP\SCU1\LIT\SCL,real,1,60,1,0.1
SCU_1.Level,_\WTP\SCU1\LIT\SCL,real,1,60,1,0.1
SCU_1.Level_mm,_\WTP\SCU1\LIT\SCL2,real,1,60,1,0.1
SCU_2.Effluent_Turbidity,_\WTP\SCU2\EFF_NTU\SCL,real,1,60,3,0.001
SCU_2.Effluent_pH,_\WTP\SCU2\EFF_PH\SCL,real,1,60,2,0.01
SCU_2.Level,_\WTP\SCU2\LIT\SCL,real,1,60,1,0.1
SCU_2.Level_mm,_\WTP\SCU2\LIT\SCL2,real,1,60,1,0.1
SCU_3.Effluent_Turbidity,_\WTP\SCU3\EFF_NTU\SCL,real,1,60,3,0.001
SCU_3.Effluent_pH,_\WTP\SCU3\EFF_PH\SCL,real,1,60,2,0.01
SCU_3.Level,_\WTP\SCU3\LIT\SCL,real,1,60,1,0.1
SCU_3.Level_mm,_\WTP\SCU3\LIT\SCL2,real,1,60,1,0.1
SCU_4.Effluent_Turbidity,_\WTP\SCU4\EFF_NTU\SCL,real,1,60,3,0.001
SCU_4.Effluent_pH,_\WTP\SCU4\EFF_PH\SCL,real,1,60,2,0.01
SCU_4.Level,_\WTP\SCU4\LIT\SCL,real,1,60,1,0.1
SCU_4.Level_mm,_\WTP\SCU4\LIT\SCL2,real,1,60,1,0.1
SCU_4.Raw_Turbidity_Max_SP,_\WTP\SCU4\RAW\AIT_RAW_NTU\AH\ALM_SP,real,1,60,3,0.001
SCU_4.Raw_Turbidity,_\WTP\SCU4\RAW\AIT_RAW_NTU\SCL,real,1,60,3,0.001
SCU_4.Raw_pH,_\WTP\SCU4\RAW\AIT_RAW_PH\SCL,real,1,60,2,0.01
SCU_4.4_SCU_PIT_SCL,_\WTP\SCU4\RAW\PIT_RAW\SCL,real,1,60,1,0.1
SCU_4.Raw_Temperature,_\WTP\SCU4\RAW\TIT_RAW\SCL,real,1,60,1,0.1
SCU_5.Effluent_Turbidity,_\WTP\SCU5\EFF_NTU\SCL,real,1,60,3,0.001
SCU_5.Effluent_pH,_\WTP\SCU5\EFF_PH\SCL,real,1,60,2,0.01
SCU_5.Level,_\WTP\SCU5\LIT\SCL,real,1,60,1,0.1
SCU_5.Sludge_Level,_\WTP\SCU5\LIT_SLUDGE\SCL,real,1,60,1,0.1
SCU_5.Raw_Turbidity,_\WTP\SCU5\RAW\AIT_RAW_NTU\SCL,real,1,60,3,0.001
SCU_5.Raw_pH,_\WTP\SCU5\RAW\AIT_RAW_PH\SCL,real,1,60,2,0.01
SCU_5.5_SCU_PIT_SCL,_\WTP\SCU5\RAW\PIT_RAW\SCL,real,1,60,1,0.1
SCU_5.Raw_Temperature,_\WTP\SCU5\RAW\TIT_RAW\SCL,real,1,60,1,0.1
ALUM.P1_Spd_Feedback,_\WWTP\CP_100\FIL\ALUM\Pump_1\YI,real,1,60,1,0.1
ALUM.P2_Spd_Feedback,_\WWTP\CP_100\FIL\ALUM\Pump_2\YI,real,1,60,1,0.1
ALUM.P3_Spd_Feedback,_\WWTP\CP_100\FIL\ALUM\Pump_2\YI,real,1,60,1,0.1
RSS_P1.CLARIFIER_SLUDGE_LVL,_\WWTP\CP_100\SEC_CLA\P1_Sludge_Level\SCL,real,1,60,1,0.1
RSS_P3.CLARIFIER_SLUDGE_LVL,_\WWTP\CP_100\SEC_CLA\P3_Sludge_Level\SCL,real,1,60,1,0.1
FEP_PIT_1.SCL,_\WWTP\FEP\FEP_PIT_1\SCL,real,1,60,1,0.1
SCB_LT_1.SCL,_\WWTP\FEP\SCB_LT_1\SCL,real,1,60,1,0.1
Airport_System.Suction_Pressure,_\DRS\Airport_BS\PIT_101\SCL,real,1,60,1,0.1
Southlands_Lift.FIT_102_Total,_\DRS\Southlands_LS\FIT_102\TOTALIZE\TOTAL,integer,1,60,0,1
Southview_Res.Start_Fill_SP,_\DRS\Southview_Reservoir\_SP\Start_Fill_SP\SP,real,1,60,1,0.1
Southview_Res.Stop_Fill_SP,_\DRS\Southview_Reservoir\_SP\Stop_Fill_SP\SP,real,1,60,1,0.1
Southview_Res.Pump_1_Runtime,_\DRS\Southview_Reservoir\BP_101\RUN_TIME\HOURS,integer,1,60,0,1
Southview_Res.Pump_2_Runtime,_\DRS\Southview_Reservoir\BP_102\RUN_TIME\HOURS,integer,1,60,0,1
Southview_Res.Pump_3_Runtime,_\DRS\Southview_Reservoir\BP_103\RUN_TIME\HOURS,integer,1,60,0,1
Southview_Res.FCV_Position,_\DRS\Southview_Reservoir\FCV_Position\ZI,real,1,60,1,0.1
Act_Carb_System.DSG_SCU1_SP,_\WTP\ActiveCarbon\_SP\DSG_SCU1_SP\SP,real,1,60,1,0.1
Act_Carb_System.DSG_SCU2_SP,_\WTP\ActiveCarbon\_SP\DSG_SCU2_SP\SP,real,1,60,1,0.1
Act_Carb_System.DSG_SCU3_SP,_\WTP\ActiveCarbon\_SP\DSG_SCU3_SP\SP,real,1,60,1,0.1
Act_Carb_System.DSG_SCU4_SP,_\WTP\ActiveCarbon\_SP\DSG_SCU4_SP\SP,real,1,60,1,0.1
Act_Carb_System.DSG_SCU5_SP,_\WTP\ActiveCarbon\_SP\DSG_SCU5_SP\SP,real,1,60,1,0.1
Alum_System.Dosage_SCU2_SP,_\WTP\AlumSystem\_SP\Dosage\SCU2_DOSAGE_SP\SP,real,1,60,1,0.1
Alum_System.Dosage_SCU3_SP,_\WTP\AlumSystem\_SP\Dosage\SCU3_DOSAGE_SP\SP,real,1,60,1,0.1
Alum_System.Dosage_SCU4_SP,_\WTP\AlumSystem\_SP\Dosage\SCU4_DOSAGE_SP\SP,real,1,60,1,0.1
Alum_System.Dosage_SCU5_SP,_\WTP\AlumSystem\_SP\Dosage\SCU5_DOSAGE_SP\SP,real,1,60,1,0.1
Alum_System.Daily_Total,_\WTP\AlumSystem\FQIT\TOTAL_TODAY,integer,1,60,0,1
Highlift_Pump_010.Discharge_Valve_Position,_\WTP\Distribution\HLPs\HIGH_LIFT_10\FCV_HLP\ZI,real,1,60,1,0.1
Highlift_Pump_012.Discharge_Valve_Position,_\WTP\Distribution\HLPs\HIGH_LIFT_12\FCV_HLP\ZI,real,1,60,1,0.1
SCU_1.Flow_Total,_\WTP\SCU1\FQIT\TOTAL,integer,1,60,0,1
SCU_2.Flow_Total,_\WTP\SCU2\FQIT\TOTAL,integer,1,60,0,1
SCU_3.Flow_Total,_\WTP\SCU3\FQIT\TOTAL,integer,1,60,0,1
SCU_4.Flow_Total,_\WTP\SCU4\FQIT\TOTAL,integer,1,60,0,1
SCU_4.Raw_pH_Max_SP,_\WTP\SCU4\RAW\AIT_RAW_PH\AH\ALM_SP,real,1,60,2,0.01
SCU_4.Raw_pH_Min_SP,_\WTP\SCU4\RAW\AIT_RAW_PH\AL\ALM_SP,real,1,60,2,0.01
RSS.RETURN_FLOW,_\WWTP\CP_100\FIL\RSS\RETURN_FLOW,real,1,60,1,0.1
RSS_P1.RUNTIME_TOT_HOURS,_\WWTP\CP_100\FIL\RSS\RSS1\RUN_TIME\HOURS,integer,1,60,0,1
RSS_P1.SC,_\WWTP\CP_100\FIL\RSS\RSS1\SC\AUTO,real,1,60,1,0.1
RSS_P1.SC_OUT,_\WWTP\CP_100\FIL\RSS\RSS1\SC\CONTR,real,1,60,1,0.1
RSS_P3.RUNTIME_TOT_HOURS,_\WWTP\CP_100\FIL\RSS\RSS3\RUN_TIME\HOURS,integer,1,60,0,1
RSS_P3.SC,_\WWTP\CP_100\FIL\RSS\RSS3\SC\AUTO,real,1,60,1,0.1
RSS_P3.SC_OUT,_\WWTP\CP_100\FIL\RSS\RSS3\SC\CONTR,real,1,60,1,0.1

1 Source Name Destination Name Data Type Scale Factor Interval (s) Precision Deadband
2 11959 AHU6_Z_SP_3000571.Value temp\new_value1 _\BACNET_HVAC\AHU6\RM_TEMP_SP\CMD real integer 1 20 100 2 0 0.01 0
3 5153 Airport_Boost_Pump_002.Runtime temp\new_value2 _\DRS\Airport_BS\P102\RUN_TIME\HOURS integer real 1 100 60 0 1 0 0.1
4 5154 Airport_Boost_Pump_004.Runtime temp\new_value3 _\DRS\Airport_BS\P104\RUN_TIME\HOURS real 1 30 60 0 1 1 0.1
5 5155 Airport_Boost_Pump_001.Runtime temp\new_value4 _\DRS\Airport_BS\P101\RUN_TIME\HOURS boolean real 1 9 60 0 1 0 0.1
6 5 Airport_Boost_Pump_003.Runtime temp\new_value5 _\DRS\Airport_BS\P103\RUN_TIME\HOURS real 1 20 60 2 1 0 0.1
7 6 AHU101_DA_TEMP_3000340.Value temp\new_value6 _\BACNET_HVAC\AHU101\SA_TEMP\SCL integer real 10 1 100 60 0 1 0 0.1
8 227 AHU6_DA_TEMP_3000446.Value temp\new_value7 _\BACNET_HVAC\AHU6\SA_TEMP\SCL real 1.5 1 100 60 2 1 0 0.1
9 8 Airport_Boost_Pump_001.Current_Demand temp\new_value8 _\DRS\Airport_BS\CT_101\SCL boolean real 1 9 60 0 1 0 0.1
10 9 Airport_Boost_Pump_002.Current_Demand temp\new_value9 _\DRS\Airport_BS\CT_102\SCL real 1 10 60 2 1 0 0.1
11 10 Airport_Boost_Pump_003.Current_Demand temp\new_value10 _\DRS\Airport_BS\CT_103\SCL integer real 1 10 60 0 1 0 0.1
12 011 Airport_Boost_Pump_004.Current_Demand temp\new_value11 _\DRS\Airport_BS\CT_104\SCL real 1 10 60 4 1 0 0.1
13 12 Airport_System.Discharge_Flow temp\new_value12 _\DRS\Airport_BS\FIT_101\SCL boolean real 1 10 60 0 1 0 0.1
14 1113 Airport_System.Total_Flow temp\new_value13 _\DRS\Airport_BS\FIT_101\TOTALIZE real 1 12 60 4 1 0 0.1
15 14 Airport_Booster.P101_SPEED temp\new_value14 _\DRS\Airport_BS\P101\YI integer real 1 13 60 0 1 0 0.1
16 1665 Airport_Booster.P102_SPEED temp\new_value15 _\DRS\Airport_BS\P102\YI real 1 14 60 4 1 0 0.1
17 16 Airport_Booster.P103_SPEED temp\new_value16 _\DRS\Airport_BS\P103\YI boolean real 1 15 60 0 1 0 0.1
18 Airport_Booster.P104_SPEED _\DRS\Airport_BS\P104\YI real 1 60 1 0.1
19 Airport_System.Discharge_Pressure _\DRS\Airport_BS\PIT_102\SCL real 1 60 1 0.1
20 Southlands_Lift.FIT_102 _\DRS\Southlands_LS\FIT_102\SCL real 1 60 1 0.1
21 Southlands_Lift.LT_127A _\DRS\Southlands_LS\LT_127A\SCL real 1 60 1 0.1
22 Southridge_Res.LIT_219_mm _\DRS\Southridge_Reservoir\LIT_219\SCL real 1 60 1 0.1
23 Southview_Res.Chlorine_Residual _\DRS\Southview_Reservoir\AIT_101\SCL real 1 60 1 0.1
24 Southview_System.Chlorine_Residual _\DRS\Southview_Reservoir\AIT_101\SCL real 1 60 1 0.1
25 Southview_Res.Pump_1_Speed_Hz _\DRS\Southview_Reservoir\BP_101\YI real 1 60 1 0.1
26 Southview_Res.Pump_2_Speed_Hz _\DRS\Southview_Reservoir\BP_102\YI real 1 60 1 0.1
27 Southview_Res.Pump_3_Speed_Hz _\DRS\Southview_Reservoir\BP_103\YI real 1 60 1 0.1
28 Southview_Res.CT_101 _\DRS\Southview_Reservoir\CT_101\SCL real 1 60 1 0.1
29 Southview_Res.CT_102 _\DRS\Southview_Reservoir\CT_102\SCL real 1 60 1 0.1
30 Southview_Res.CT_103 _\DRS\Southview_Reservoir\CT_103\SCL real 1 60 1 0.1
31 Southview_Res.FIT_101 _\DRS\Southview_Reservoir\FIT_101\SCL real 1 60 1 0.1
32 Southview_System.Inlet_Flow _\DRS\Southview_Reservoir\FIT_101\SCL real 1 60 1 0.1
33 Southview_Res.FIT_102 _\DRS\Southview_Reservoir\FIT_102\SCL real 1 60 1 0.1
34 Southview_System.Discharge_Flow _\DRS\Southview_Reservoir\FIT_102\SCL real 1 60 1 0.1
35 Southview_Res.LT_101 _\DRS\Southview_Reservoir\LIT_101\SCL real 1 60 1 0.1
36 Southview_System.Level _\DRS\Southview_Reservoir\LIT_101\SCL real 1 60 1 0.1
37 Southview_Res.LT_101_mm _\DRS\Southview_Reservoir\LIT_101\SCL2 real 1 60 1 0.1
38 Southview_System.Level_m _\DRS\Southview_Reservoir\LIT_101\SCL2 real 1 60 1 0.1
39 Southview_Res.PT_101 _\DRS\Southview_Reservoir\PIT_101\SCL real 1 60 1 0.1
40 Southview_System.Inlet_Pressure _\DRS\Southview_Reservoir\PIT_101\SCL real 1 60 1 0.1
41 Southview_Res.PT_102 _\DRS\Southview_Reservoir\PIT_102\SCL real 1 60 1 0.1
42 Southview_System.Discharge_Pressure _\DRS\Southview_Reservoir\PIT_102\SCL real 1 60 1 0.1
43 AIT_DU_8638.SCL _\WTP\ChlorScrubber\AIT_DU_8638\SCL real 1 60 1 0.1
44 AIT_DU_8639.SCL _\WTP\ChlorScrubber\AIT_DU_8639\SCL real 1 60 1 0.1
45 AIT_DU_8640.SCL _\WTP\ChlorScrubber\AIT_DU_8640\SCL real 1 60 1 0.1
46 AIT_DU_8641.SCL _\WTP\ChlorScrubber\AIT_DU_8641\SCL real 1 60 1 0.1
47 AIT_DU_8642.SCL _\WTP\ChlorScrubber\AIT_DU_8642\SCL real 1 60 1 0.1
48 AIT_DU_8643.SCL _\WTP\ChlorScrubber\AIT_DU_8643\SCL real 1 60 1 0.1
49 AIT_DU_8644.SCL _\WTP\ChlorScrubber\AIT_DU_8644\SCL real 1 60 1 0.1
50 AIT_DU_8645.SCL _\WTP\ChlorScrubber\AIT_DU_8645\SCL real 1 60 1 0.1
51 AIT8809.SCL _\WTP\Dechlor\Tank_12\AIT8809\SCL real 1 60 1 0.1
52 AIT8811.SCL _\WTP\Dechlor\Tank_12\AIT8811\SCL real 1 60 1 0.1
53 AIT8803.SCL _\WTP\Dechlor\Tank_3\AIT8803\SCL real 1 60 1 0.1
54 AIT8805.SCL _\WTP\Dechlor\Tank_3\AIT8805\SCL real 1 60 1 0.1
55 SCU_1.Effluent_Turbidity _\WTP\SCU1\EFF_NTU\SCL real 1 60 3 0.001
56 SCU_1.Effluent_pH _\WTP\SCU1\EFF_PH\SCL real 1 60 2 0.01
57 Act_Carb_Tank_001.Tank_Level_L _\WTP\SCU1\LIT\SCL real 1 60 1 0.1
58 SCU_1.Level _\WTP\SCU1\LIT\SCL real 1 60 1 0.1
59 SCU_1.Level_mm _\WTP\SCU1\LIT\SCL2 real 1 60 1 0.1
60 SCU_2.Effluent_Turbidity _\WTP\SCU2\EFF_NTU\SCL real 1 60 3 0.001
61 SCU_2.Effluent_pH _\WTP\SCU2\EFF_PH\SCL real 1 60 2 0.01
62 SCU_2.Level _\WTP\SCU2\LIT\SCL real 1 60 1 0.1
63 SCU_2.Level_mm _\WTP\SCU2\LIT\SCL2 real 1 60 1 0.1
64 SCU_3.Effluent_Turbidity _\WTP\SCU3\EFF_NTU\SCL real 1 60 3 0.001
65 SCU_3.Effluent_pH _\WTP\SCU3\EFF_PH\SCL real 1 60 2 0.01
66 SCU_3.Level _\WTP\SCU3\LIT\SCL real 1 60 1 0.1
67 SCU_3.Level_mm _\WTP\SCU3\LIT\SCL2 real 1 60 1 0.1
68 SCU_4.Effluent_Turbidity _\WTP\SCU4\EFF_NTU\SCL real 1 60 3 0.001
69 SCU_4.Effluent_pH _\WTP\SCU4\EFF_PH\SCL real 1 60 2 0.01
70 SCU_4.Level _\WTP\SCU4\LIT\SCL real 1 60 1 0.1
71 SCU_4.Level_mm _\WTP\SCU4\LIT\SCL2 real 1 60 1 0.1
72 SCU_4.Raw_Turbidity_Max_SP _\WTP\SCU4\RAW\AIT_RAW_NTU\AH\ALM_SP real 1 60 3 0.001
73 SCU_4.Raw_Turbidity _\WTP\SCU4\RAW\AIT_RAW_NTU\SCL real 1 60 3 0.001
74 SCU_4.Raw_pH _\WTP\SCU4\RAW\AIT_RAW_PH\SCL real 1 60 2 0.01
75 SCU_4.4_SCU_PIT_SCL _\WTP\SCU4\RAW\PIT_RAW\SCL real 1 60 1 0.1
76 SCU_4.Raw_Temperature _\WTP\SCU4\RAW\TIT_RAW\SCL real 1 60 1 0.1
77 SCU_5.Effluent_Turbidity _\WTP\SCU5\EFF_NTU\SCL real 1 60 3 0.001
78 SCU_5.Effluent_pH _\WTP\SCU5\EFF_PH\SCL real 1 60 2 0.01
79 SCU_5.Level _\WTP\SCU5\LIT\SCL real 1 60 1 0.1
80 SCU_5.Sludge_Level _\WTP\SCU5\LIT_SLUDGE\SCL real 1 60 1 0.1
81 SCU_5.Raw_Turbidity _\WTP\SCU5\RAW\AIT_RAW_NTU\SCL real 1 60 3 0.001
82 SCU_5.Raw_pH _\WTP\SCU5\RAW\AIT_RAW_PH\SCL real 1 60 2 0.01
83 SCU_5.5_SCU_PIT_SCL _\WTP\SCU5\RAW\PIT_RAW\SCL real 1 60 1 0.1
84 SCU_5.Raw_Temperature _\WTP\SCU5\RAW\TIT_RAW\SCL real 1 60 1 0.1
85 ALUM.P1_Spd_Feedback _\WWTP\CP_100\FIL\ALUM\Pump_1\YI real 1 60 1 0.1
86 ALUM.P2_Spd_Feedback _\WWTP\CP_100\FIL\ALUM\Pump_2\YI real 1 60 1 0.1
87 ALUM.P3_Spd_Feedback _\WWTP\CP_100\FIL\ALUM\Pump_2\YI real 1 60 1 0.1
88 RSS_P1.CLARIFIER_SLUDGE_LVL _\WWTP\CP_100\SEC_CLA\P1_Sludge_Level\SCL real 1 60 1 0.1
89 RSS_P3.CLARIFIER_SLUDGE_LVL _\WWTP\CP_100\SEC_CLA\P3_Sludge_Level\SCL real 1 60 1 0.1
90 FEP_PIT_1.SCL _\WWTP\FEP\FEP_PIT_1\SCL real 1 60 1 0.1
91 SCB_LT_1.SCL _\WWTP\FEP\SCB_LT_1\SCL real 1 60 1 0.1
92 Airport_System.Suction_Pressure _\DRS\Airport_BS\PIT_101\SCL real 1 60 1 0.1
93 Southlands_Lift.FIT_102_Total _\DRS\Southlands_LS\FIT_102\TOTALIZE\TOTAL integer 1 60 0 1
94 Southview_Res.Start_Fill_SP _\DRS\Southview_Reservoir\_SP\Start_Fill_SP\SP real 1 60 1 0.1
95 Southview_Res.Stop_Fill_SP _\DRS\Southview_Reservoir\_SP\Stop_Fill_SP\SP real 1 60 1 0.1
96 Southview_Res.Pump_1_Runtime _\DRS\Southview_Reservoir\BP_101\RUN_TIME\HOURS integer 1 60 0 1
97 Southview_Res.Pump_2_Runtime _\DRS\Southview_Reservoir\BP_102\RUN_TIME\HOURS integer 1 60 0 1
98 Southview_Res.Pump_3_Runtime _\DRS\Southview_Reservoir\BP_103\RUN_TIME\HOURS integer 1 60 0 1
99 Southview_Res.FCV_Position _\DRS\Southview_Reservoir\FCV_Position\ZI real 1 60 1 0.1
100 Act_Carb_System.DSG_SCU1_SP _\WTP\ActiveCarbon\_SP\DSG_SCU1_SP\SP real 1 60 1 0.1
101 Act_Carb_System.DSG_SCU2_SP _\WTP\ActiveCarbon\_SP\DSG_SCU2_SP\SP real 1 60 1 0.1
102 Act_Carb_System.DSG_SCU3_SP _\WTP\ActiveCarbon\_SP\DSG_SCU3_SP\SP real 1 60 1 0.1
103 Act_Carb_System.DSG_SCU4_SP _\WTP\ActiveCarbon\_SP\DSG_SCU4_SP\SP real 1 60 1 0.1
104 Act_Carb_System.DSG_SCU5_SP _\WTP\ActiveCarbon\_SP\DSG_SCU5_SP\SP real 1 60 1 0.1
105 Alum_System.Dosage_SCU2_SP _\WTP\AlumSystem\_SP\Dosage\SCU2_DOSAGE_SP\SP real 1 60 1 0.1
106 Alum_System.Dosage_SCU3_SP _\WTP\AlumSystem\_SP\Dosage\SCU3_DOSAGE_SP\SP real 1 60 1 0.1
107 Alum_System.Dosage_SCU4_SP _\WTP\AlumSystem\_SP\Dosage\SCU4_DOSAGE_SP\SP real 1 60 1 0.1
108 Alum_System.Dosage_SCU5_SP _\WTP\AlumSystem\_SP\Dosage\SCU5_DOSAGE_SP\SP real 1 60 1 0.1
109 Alum_System.Daily_Total _\WTP\AlumSystem\FQIT\TOTAL_TODAY integer 1 60 0 1
110 Highlift_Pump_010.Discharge_Valve_Position _\WTP\Distribution\HLPs\HIGH_LIFT_10\FCV_HLP\ZI real 1 60 1 0.1
111 Highlift_Pump_012.Discharge_Valve_Position _\WTP\Distribution\HLPs\HIGH_LIFT_12\FCV_HLP\ZI real 1 60 1 0.1
112 SCU_1.Flow_Total _\WTP\SCU1\FQIT\TOTAL integer 1 60 0 1
113 SCU_2.Flow_Total _\WTP\SCU2\FQIT\TOTAL integer 1 60 0 1
114 SCU_3.Flow_Total _\WTP\SCU3\FQIT\TOTAL integer 1 60 0 1
115 SCU_4.Flow_Total _\WTP\SCU4\FQIT\TOTAL integer 1 60 0 1
116 SCU_4.Raw_pH_Max_SP _\WTP\SCU4\RAW\AIT_RAW_PH\AH\ALM_SP real 1 60 2 0.01
117 SCU_4.Raw_pH_Min_SP _\WTP\SCU4\RAW\AIT_RAW_PH\AL\ALM_SP real 1 60 2 0.01
118 RSS.RETURN_FLOW _\WWTP\CP_100\FIL\RSS\RETURN_FLOW real 1 60 1 0.1
119 RSS_P1.RUNTIME_TOT_HOURS _\WWTP\CP_100\FIL\RSS\RSS1\RUN_TIME\HOURS integer 1 60 0 1
120 RSS_P1.SC _\WWTP\CP_100\FIL\RSS\RSS1\SC\AUTO real 1 60 1 0.1
121 RSS_P1.SC_OUT _\WWTP\CP_100\FIL\RSS\RSS1\SC\CONTR real 1 60 1 0.1
122 RSS_P3.RUNTIME_TOT_HOURS _\WWTP\CP_100\FIL\RSS\RSS3\RUN_TIME\HOURS integer 1 60 0 1
123 RSS_P3.SC _\WWTP\CP_100\FIL\RSS\RSS3\SC\AUTO real 1 60 1 0.1
124 RSS_P3.SC_OUT _\WWTP\CP_100\FIL\RSS\RSS3\SC\CONTR real 1 60 1 0.1

34
main.py
View File

@ -4,8 +4,10 @@ import sys
import os import os
import requests import requests
import pyodbc import pyodbc
import pytz
import subprocess import subprocess
import time import time
from typing import List, Union from typing import List, Union
from datetime import datetime, timezone, timedelta from datetime import datetime, timezone, timedelta
@ -48,9 +50,8 @@ class HistoricalTag:
# aveva_query() # aveva_query()
# ---------------------- # ----------------------
#
def aveva_query(historical_tags: List[HistoricalTag], start_time: datetime, end_time: datetime): def aveva_query(historical_tags: List[HistoricalTag], start_time: datetime, end_time: datetime):
current_start_time = start_time current_start_time = start_time
current_end_time = start_time current_end_time = start_time
@ -65,9 +66,12 @@ def aveva_query(historical_tags: List[HistoricalTag], start_time: datetime, end_
else: else:
current_end_time = current_end_time + timedelta(days=92) current_end_time = current_end_time + timedelta(days=92)
tag_mappings.append(aveva_query_date(historical_tags, current_start_time, current_end_time)) query_mappings = aveva_query_date(historical_tags, current_start_time, current_end_time)
current_start_time = current_start_time + timedelta(days=92)
for mapping in query_mappings:
tag_mappings.append(mapping)
current_start_time = current_start_time + timedelta(days=92)
write_tagmapping_to_file(dir_path + "TagMapping.csv", tag_mappings) write_tagmapping_to_file(dir_path + "TagMapping.csv", tag_mappings)
@ -101,8 +105,8 @@ def aveva_query_date(historical_tags: List[HistoricalTag], start_time: datetime,
ft_end_time = '"' + str(end_time.astimezone(timezone.utc)) + '"' ft_end_time = '"' + str(end_time.astimezone(timezone.utc)) + '"'
ft_tag_str = '[' + str(tag.name_source) + ']' ft_tag_str = '[' + str(tag.name_source) + ']'
query = f"SELECT * FROM OpenQuery(INSQL, 'SELECT DateTime, {ft_tag_str} FROM WideHistory WHERE DateTime >=" + \ query = f"SELECT * FROM OpenQuery(INSQL, 'SELECT DateTime, {ft_tag_str} FROM WideHistory WHERE DateTime >= " + \
f"{ft_start_time} AND DateTime <= {ft_end_time} AND wwRetrievalMode = {retrieval_mode}" + \ f"{ft_start_time} AND DateTime <= {ft_end_time} AND wwRetrievalMode = {retrieval_mode} " + \
f"AND wwResolution = {str(tag.interval * 1000)}')" f"AND wwResolution = {str(tag.interval * 1000)}')"
cursor = connection.cursor() cursor = connection.cursor()
@ -406,7 +410,7 @@ def postprocess_values(values: List[Union[int, float, None]]):
def prepare_file_for_tag(tag: HistoricalTag, values: List[Union[int, float, None]], dir_path: str, current_end_time: datetime, append=False) -> str: def prepare_file_for_tag(tag: HistoricalTag, values: List[Union[int, float, None]], dir_path: str, current_end_time: datetime, append=False) -> str:
if values is None: if len(values) == 0:
print("No values found") print("No values found")
return "" return ""
else: else:
@ -468,6 +472,8 @@ def write_tagmapping_to_file(output_file: str, tag_mappings: List[str]):
with open(output_file, mode='a', newline='', encoding='utf-8') as csvfile: with open(output_file, mode='a', newline='', encoding='utf-8') as csvfile:
csv_writer = csv.writer(csvfile) csv_writer = csv.writer(csvfile)
print(len(tag_mappings))
for mapping in tag_mappings: for mapping in tag_mappings:
csv_writer.writerow(mapping) csv_writer.writerow(mapping)
@ -490,7 +496,7 @@ def write_values_to_file(output_file: str, values: List[Union[int, float, None]]
for value_pair in values: for value_pair in values:
timestamp, value = value_pair timestamp, value = value_pair
if value is not None: if value is not None:
utc_dt = datetime.utcfromtimestamp(timestamp) utc_dt = datetime.fromtimestamp(timestamp, config_tz)
formatted_timestamp = utc_dt.strftime( formatted_timestamp = utc_dt.strftime(
'%Y-%m-%d %H:%M:%S.%f')[:-3] '%Y-%m-%d %H:%M:%S.%f')[:-3]
csv_writer.writerow([formatted_timestamp, value]) csv_writer.writerow([formatted_timestamp, value])
@ -504,9 +510,9 @@ def write_values_to_file(output_file: str, values: List[Union[int, float, None]]
def weeks_since_date(timestamp, date=(1601, 1, 1)): def weeks_since_date(timestamp, date=(1601, 1, 1)):
dt = datetime.utcfromtimestamp(timestamp) utc_dt = datetime.fromtimestamp(timestamp, config_tz)
start_date = datetime(*date) start_date = datetime(*date)
delta = dt - start_date delta = utc_dt - start_date
weeks = delta.days // 7 weeks = delta.days // 7
return weeks return weeks
@ -528,7 +534,7 @@ config = toml.load("setup.toml")
tags_path = config['system']['tags_path'] tags_path = config['system']['tags_path']
output_path = config['system']['output_path'] output_path = config['system']['output_path']
system_timezone = config['system']['system_timezone'] config_timezone = config['system']['system_timezone']
application_user = config['user']['application_user'] application_user = config['user']['application_user']
application_pass = config['user']['application_pass'] application_pass = config['user']['application_pass']
@ -540,12 +546,14 @@ if len(sys.argv) == 4:
year, month, day = map(int, str(sys.argv[3]).split("-")) year, month, day = map(int, str(sys.argv[3]).split("-"))
end_time = datetime(year, month, day, 0, 0, 0) end_time = datetime(year, month, day, 0, 0, 0)
config_tz = pytz.timezone(config_timezone)
print("") print("")
historical_tags = read_tags(tags_path) historical_tags = read_tags(tags_path)
for tag in historical_tags: #for tag in historical_tags:
print(tag) # print(tag)
if query_type == "VTScada": if query_type == "VTScada":
print_text('VTScada Data Query') print_text('VTScada Data Query')

View File

@ -8,7 +8,7 @@ output_path = "output\\"
system_timezone = "Canada/Mountain" system_timezone = "Canada/Mountain"
[aveva] [aveva]
server_name = "HIST002" server_name = "192.168.1.41"
database_name = "Runtime" database_name = "Runtime"
[vtscada] [vtscada]