                  File: massage.txt      by M Gunter

This file outlines the manipulation (or "massaging") of data collected
on site at the Breamlea wind generator. The file logger.txt in this
directory is a listing of the "schedule" (or program) which configures
the DataTaker DT50 data logger to record data in its RAM. Normally this
data would be periodically downloaded to a PC using a serial interface and 
proprietary software. Just to be a bit different, I decided to
interrogate the logger using "Telix" running a simple SALT script. This
was set up on COM1 of an old Toshiba T1100 which a relative happened to
own (subsequently donated to the ATA). As luck would have it,
a 2400 bps internal modem was available for that same computer. When 
plugged in, it appears on COM2. Thus it is possible to run Telix as a
sequence of calls from a DOS batch file, each one configured to talk 
either to the logger or the modem. The batch file loops back on itself
every 4 days, so on the fourth day when it exits the "daily" script for
the fourth time, it rearranges the on-board backup files before getting
telix (configured with the "determ.cnf" file) to unload the memory of
the data logger to a telix capture file (good old "telix.cap"!). So here
is the listing of "breamlea.bat":

:START
A:
TELIX SDAILY.SLC
TELIX SDAILY.SLC
TELIX SDAILY.SLC
TELIX SDAILY.SLC
B:
IF EXIST TELIXCAP.BAK DEL TELIXCAP.BAK
IF EXIST TELIX.CAP REN TELIX.CAP TELIXCAP.BAK
A:
IF EXIST NEWDATA.BAK DEL NEWDATA.BAK
IF EXIST NEWDATA.ZIP REN NEWDATA.ZIP *.BAK
TELIX CDETERM.CNF SUNLOAD.SLC
TELIX SMICHAEL.SLC
GOTO START

The next listing is the timing file "daily.slc" which uses the computer's
clock to jump to the next line of the batch file once daily. If you're
familiar with the Telix script language SALT you will know that the
*.slt files must be compiled using cs.exe to *.slc files before telix can
interpret them. All the scripts here are *.slt versions.

main()
{
cputs_tr("~~~+++~~~ATH0^M");
start:
 delay_scr(100);
 if (thour(curtime())==20 && tmin(curtime())==50)
  exittelix(0,1);
goto start;
}

The next script, unload.slc prompts the data logger to send a status
report and unload its most recent data: that is done by the string
in the "cputs_tr" line. Remember that Telix is talking to the logger
on COM1 at this time, not to the modem on COM2, because it was run
using the "determ.cnf" configuration file which specifies which COM
port, baud rate, etc


main()
{
capture("b:\telix.cap");
delay_scr(20);
cputs_tr("^M~/U^M~STATUS^M~~~/u^M~U(LAST)^M");
delay_scr(3000);
capture("*CLOSE*");
exittelix(0,1);
}

The second last line of the batch file contains telix in default config-
uration (telix.cnf) talking to the modem using the "michael.slc" script
which rings my computer at home:

main()
{
 run("pkzip.exe","newdata b:\telix.cap");
 cputs_tr("ATDT0393767515^M");
 delay_scr(80);
 cputs_tr("~~~+++~~~ATH0^M");
 delay_scr(1200);
 dial("m0393767515",3);
 send('Z',"newdata.zip");
 cputs_tr("~~~+++~~~ATH0^M");
 exittelix(0,1);
 }

Looking at the above script as I write this note (finally documenting
something I patched together over 7 months ago!), I cannot imagine why
I decided to compress the capture file by running pkzip as a "shell"
out of telix. It would be much more logical to run it BEFORE invoking
the modem-talking telix program i.e. straight after the file captured
from the data logger is written to the b: floppy (yes - this setup
doesn't even need a hard disk!) Anyhow, it works! Note that it gives
a short warning ring, and then waits 120 seconds before ringing again
to send the zipped data as a Zmodem transfer. We had terrible trouble
as complete novices getting THAT to work, until it finally dawned on
us that you MUST disable xon/xoff flow control when sending zipped
files, as they are liberally sprinkled throughout with ASCII characters
17 and 19 (you guessed it - the xon/xoff characters!!) No wonder the
file transfers kept crashing after about half a second!

Meanwhile, back at home, roused by the first ring, I have staggered out
of bed, put the kettle on for a cuppa and fired up Telix in auto-Zmodem-
download-mode. Within a few seconds of the second call, the file is
safely tucked away on my home computer's hard disk, awaiting the assault
of dBaseIII+

But wait ! There's more !  (preparation, that is)  First I unzip the
data file and decapitate it: that is remove the status report "header".
This is a fiddly manual operation, but fairly easily achieved using the
"gather" function of Xtree Gold. If you have never had cause to use it,
it is a subfunction of file viewing: text is highlighted or "gathered"
and then appended to any file name you specify - I call it "new.log"
and it consists of all of the capture file except the status report at
the top.

Now for the big finale !  Am I the only computer Luddite still using
dBaseIII+ in the whole universe? Any dBase "Trekkies" out there can
Email me at mickgg@suburbia.net for some old fashioned dBase banter
such as "go bott", "appe from" and my favourite "set rela to". The 
following routine is actually a concatenation of three smaller routines
that evolved over an 8 month period, and my word, DOESN'T IT SHOW !
Who cares - it does what I want semi-automatically, and just imagine
how much cutting and pasting would be required to achieve one tenth
of the result using Excel (I know nothing of newer generation databases
so I cannot comment about them - someone enlighten me please)

!copy newdt.dbf *.bak
set talk on
set stat on
set date brit
use newdt
go bott
mrecno=recno()
append from new.log deli with blan
goto mrecno+1
* the next line creates a character field which sorts "alphabetically" by date and time
repl datetime with righ(chardate,4)+subs(chardate,4,2)+left(chardate,2)+left(time,2)+subs(time,4,2)+righ(time,2) whil .t.
set filter to time='00:00:00'
* the data logger's main fault is the way it records the date and time: any
* summing function which is logged at midnight on day x will actually carry
* the date of day x+1. The next line converts all 0000 hrs loggings on
* day n to be recorded as 2400hrs on day n-1. Try doing that with Excel !
repl datetime with '19'+righ(dtoc(ctod(chardate)-1),2)+subs(dtoc(ctod(chardate)-1),4,2)+left(dtoc(ctod(chardate)-1),2)+'2400' all
set filt to
* you still have a local backup in the zip file, so............
!del telix.cap
!del new.log
goto mrecno+1
repl no_shadow with av_ws whil .t.
goto mrecno
set filt to kwh>0 .and. ((dirn>0 .and. dirn<90) .or. dirn>330)
skip
* the following polynomial expression is used to calculate (from energy
* data) what the anemometer would have been reading if it was not in
* the shadow of the wind generator's rotor. The anemometer is directly
* behind the rotor when wind is from the northeast (about 30 degrees)
repl no_shadow with .0006*(kwh**3)-.0251*(kwh**2)+.5584*kwh+3.5256 whil .t.
set filter to av_ws>no_shadow
goto mrecno
skip
* this next line is statistically invalid (??)
repl no_shadow with av_ws whil .not. eof()
set filt to ((dirn>0 .and. dirn<90) .or. dirn>330)
?'Winds at 22 metres (raw & corrected) are as follows'
?'for the shadowed sector 330 deg to 90 deg.'
goto mrecno
skip
set talk on
aver av_ws,no_shadow for ((dirn>0 .and. dirn<90) .or. dirn>330) whil .not. eof()
?'Winds at 22 Metres (raw and corrected) for directions 90 to 330 degrees:'
set filt to dirn>89 .and. dirn<331
goto mrecno
skip
aver av_ws,no_shadow whil .not. eof()
wait
set safety off
set filt to
inde on righ(datetime,4) to time
sele 2
use timeoday
if .not. bof()
zap
endif
sele 1
go top
do whil .not. eof()
Mtime=righ(datetime,4)
aver av_ws,no_shadow,2*kwh to Mraw,Mcorr,Mkw whil righ(datetime,4)=Mtime
sele 2
appe blan
repl time with Mtime,raw_ws with Mraw,no_shadow with Mcorr,kw with Mkw
sele 1
enddo
use
sele 2
go top
brow
clos all
set safety on
retu


Well that's about it. The rest is just your usual sorting and charting,
for which I use Excel. The results of that can be seen in some of the
gif charts such as history.gif and dec95.gif. Shrinking the charts for
display on the Web would be to chronicle yet another tedious operation
(using the software that I have to hand, anyway). A precis of how I do
it is on the chart demo.gif                          M Gunter 2-2-1996
