%
% Both "article" and "report" styles can be used, but article is better.
% This should work with most LaTeX 2.xx and LaTeX2e systems. The only
% external package is psfig.sty.
%
% To use this sample you need:
%    Latex2e (of cource)
%
% and files:
%
%    article.sty (usually comes with LateX)
%    psfig.sty   (usually comes with LateX)
%    ----------------------------------------------
%    sampleGSS.tex (this file - if you can read it, you have it)
%    gssl2e.sty    (specially made style for EUREDIT)
%    GSSlogo.ps    (ONS logo in postscript format)
%    chart.ps      (example postscript image)
%
% Then just run
%   > latex sampleGSS.tex
%   > latex sampleGSS.tex (two times because of the bibliography)
%   > dvips -t a4 sampleGSS.dvi
%
% now you should have the postscript document: 
%    sampleGSS.ps
%
% -----------------------------------------------------------------------
%  Here it starts
% #######################################################################
%
%
\documentclass[twoside]{article}
%
% You may use onesided documents as well:
%
% \documentclass{article}
%
% This version of the "gssl2e.sty" file uses "psfig" package for 
% postscript images. It is easy to change, if needed. Only difference 
% between the packages is the \psfig{figure=#1.ps,width=#2} command in
% gssl2e.sty file. 
%
%
\usepackage{graphicx} % grafiikkaa varten
%
\usepackage{psfig}
%
% ---------------------------------------------------------------------
% OBS! YOU MUST CHANGE THESE !!!
% !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
%
%  put here your report declarations - in this order.
%
\def\HeadingText{Author Guidelines} 
\def\EureditRapNo{XXX.XX}
%
\def\Re#1{{\rm I}\!{\rm R}^{#1}}
\def\SSet#1{{\mathcal D}_{#1}}
\def\Vx{ {\bf v'}({\bf x}) }
\def\Xv{ {\bf x'}({\bf v}) }
\def\Xvi#1{ {\bf x'}({\bf v}_#1) }
\def\Xvv{ {\bf x'}({\bf v''}) }
\def\vx#1{{v_#1}({\bf x}) }
\def\xv#1{{x_#1}({\bf v}) }
\def\pf{  f_X({\bf x}) }
\def\pv{ f_V({\bf v}) }
\def\hv{{\bf H}({\bf v}) }
\def\hx{{\bf H}({\bf x}) }
\def\hz{{\bf H}({\bf z}) }
\def\gh{ \sigma }
\def\Ds{\displaystyle}
\def\DS#1{\displaystyle #1}

\def\HH#1{ {\bf H} ({ #1 }) }
\def\Hv#1{ {\bf H} ({{\bf v} - { #1 }} ) }
\def\Hvh#1{ {\bf H} ({{\bf v} - { #1 }\over h} ) }
\def\Hx#1{ {\bf H} ({{\bf x} - { #1 }} ) }
\def\Hz#1{ {\bf H} ({{\bf z} - { #1 }} ) }
\def\pve#1{ \hat{f}_V({\bf v} \ | \ #1 ) }
\def\pfe#1{ f_X({\bf x} \ | \ #1 ) }
\def\pfg#1{ \hat{f}({\bf x} \ | \ #1 ) }
\def\pflg#1#2{ \hat{p}^{#2}({\bf x} \ | \ #1 ) }
% ---------------------------------------------
\def\HeadingText{JyU Software (1. Nov. 2002)} 
\def\EureditRapNo{D4.5\&D5.5}
\def\R{{\rm I}{\!}{\rm R}}
\def\P#1{{\rm Pr}\{{#1}\}} 
\def\E#1{{\rm E}[{#1}]} 

%
\usepackage{./jyrep}
%
% It seems that we have two kinds of report numbers?
%
\GSSRapNo{1}
%
% ---------------------------------------------------------------------
%  Sample text begins here...
%

\title{A DLL-library for the TS-SOM Based \\
Editing and Imputation Software.\\
}

\author{ Ismo Horppu - University of Jyv\"askyl\"a\\
         Pasi P.~Koikkalainen - University of Jyv\"askyl\"a}




\begin{document}

\maketitle

\tableofcontents
\newpage



% --------------------------------------------------------------------------


\summary{
%
%
This document specifies a programming interface
for a library version of the TS-SOM based editing and 
imputation system. The software is implemented 
with the Neural Data Analysis (NDA)\footnote{
\copyright NDA Software is under Copyright of the Laboratory of
Data Analysis/Univeristy of Jyv\"askyl\"a, Finland} 
software platform, including a NDA library, example macros 
and a sample C-program. 
\\ \\
The interface has been tested and used under several  
host programs, including  
Delph/Kylix, VisualBasic, Exel, Matlab, and X11/R4. 
This document explains how NDA routines
can be used from the Microsoft Visual C programming language.
\\ \\
The document is organized as follows.
Section \ref{sec:intro} is a brief introduction to NDA. 
In section \ref{sec:data_transfer}
the principles of data transfer  between NDA and an external programs 
are described. Actual procedures for editing and 
imputation are explained  in section 
\ref{sec:tssom_edit_and_imputation}, including definitions
for data and algorithm specific parameters.
System spefific files are described in the section
\ref{sec:nda_files}, including installing instructions and
NDA DLL interface on-line documentation.
Finally  an example of a Visual C project is described in the 
appendix \ref{sec:vc_example}. 
%
% List of NDA DLL functions with brief
% descriptions are given in section \ref{sec:dll_functions}.
}

\section{Introduction to Neural Data Analysis DLL Library (NDA-DLL)}

\label{sec:intro}

The NDA is a software that implements 
applications. Is consists of five layers: 

\begin{description}
\item A data layer, which provides memory management routines for
the NDA data structures. It also implements several error prevention
and grabage collection mechanisms.
\item Operation layer, which is a collecion of commands
and algorithms. These are groupped under several modules that
can be compiled and included into the NDA distribution separately.
\item Interface layer, which provides function entry points
for NDA routines. It also allows one to access NDA data structures
and to execute application level macros.
\item Application layer, which is a macro language
intepreter for NDA commands. All the functionality of NDA applications
is implemented this way.
\item End-user layer, which implements the user interface or
host programs for the NDA application.
\end{description}



The NDA/DLL is a library that implements the first four layer
of the NDA software. Thus it allows the execution of NDA macros
and the manipulation of NDA data structures from external programs. 
Typical usage of the NDA/DLL starts with the uploading of data into
NDA namespace, which is followed by the execution of NDA macros.
Finally the results are loaded from the NDA to the host application.
This is summarized in Fig. \ref{fig:exe}.

\begin{figure}[htb]
\begin{enumerate}
        \item upload data (= input) to NDA's namespace,
        \item run NDA macro command(s)
        \item download data (= output) from NDA's namespace.
\end{enumerate}
\caption{ \label{fig:exe} The steps of the usage of
 NDA/DLL from application programs.}
\end{figure}

The Windows compatible NDA/DLL provides a set of functions 
that operate with three types of data
values: float (32-bit), signed integer (32-bit), and
string (8-bit characters).


\section{Transfering data}

\label{sec:data_transfer}

The data uploading and loading routines for the NDA/DLL
interface are introduced in this section.

\subsection{\bf Transfering data to the NDA namespace}

In the NDA terminology {\tt data} is a structure of one or more
{\ tt field}s (variables). Since also {\tt field}s are NDA 
structures the   data transfer from an external program is done 
in two steps.


\begin{enumerate}
\item
 Before data can be
 transferred to the namespace, a data frame must be created
 using function {\tt nda\_create\_new\_dataframe}.
\item
 Next all
variables must be inserted into the created data frame. The
insertion can be done using functions\newline
{\tt nda\_insert\_ifield},\newline
{\tt nda\_insert\_ffield}, or \newline
{\tt nda\_insert\_sfield}. 
\end{enumerate}

The uploading functions correspond to 32-bit signed integer, 32-bit
float and 8-bit string types, respectively.
The  functions make a copy
of the given data, and thus the original data set can be 
deallocated after the call (if possible).

{\bf NOTES:}

\begin{description}
\item[a)] data frames and variables (= fields) names in
the NDA's namespace may not contain special characters, like space
or "/" etc, therefore it is best to use simple names.
%
\item[b)] old data
frame named {\tt "data"} can be deleted using \newline
{\tt run\_nda\_command("rm data");}
\end{description}

\subsubsection{Uploading example: 32-bit float data transfer to the NDA namespace}

Example 1 creates a new data frame named {\sl data}, and inserts a
32-bit float variable into NDA's namespace.

\begin{verbatim}
int Status;                            /* function call errorcode */
float *myVar;                          /* pointer to data, which
                                          has 1000 entries)       */

/* allocate and fill myVar here */

Status = nda_create_new_frame("data"); /* create new data frame   */
if (Status != OPERATION_OK) return -1; /* failure? */

/* create variable "var1" into data frame "data" */
Status = nda_insert_ffield("data", "var1", 1000, myVar);
if (Status != OPERATION_OK) return -1; /* failure? */
\end{verbatim}



\subsection{\bf Getting data from the namespace}

 The data transfer from the NDA namespace is done using function

{\tt nda\_get\_field\_ptr}.


The information is assingned to the position, 
provided by a given {\tt (void **)} pointer. 
For integers and floats it expect 
{\tt int*} and {\tt float*} types,
whereas for a string the data type is  {\tt char**}. 

\subsubsection{Downloading example from the NDA}

The following retrieves
pointer to 32-bit float variable which was created in the previous example

\begin{verbatim}
int Status;                             /* function call errorcode */
float *fvect;                           /* float data */
long records;                           /* amount of records */

/* retrieve first variable, whose index is 0, in data frame data */
Status = nda_get_field_ptr("data", 0, (void **)&fvect, &records);
if (Status != OPERATION_OK) return -1; /* failure? */

/* access data via fvect ... */
\end{verbatim}


\section{The execution of TS-SOM based editing and imputation routines}

\label{sec:tssom_edit_and_imputation}

The NDA macros for editing and imputation include several options, 
which define the functionality of the procedures as well as the given 
output data sets.

For a given incomplete and erroneous data set one can do either:
\begin{description}
\item[a)] Imputation, which gives as an output a completed 
data set
\item[b)] Editing (outlier detection), which gives are matrix of
error probabilities.
\item[c)] Both editing and imputation, in which case bout a complete
data and the probabilitys matrix is given as an output. 
\end{description}

To make the use of the editing and imputation routines simple, 
two pre-programmed macros are provided with the NDA/DLL package:

\begin{description}
\item[1. {\tt impute.cmd}] is a NDA marco to impute missing values in
a given data set.
\item[2. {\tt edit.cmd}] is a NDA macro to edit (detect outliers)
of a given data set. This routine can also do imputation of both the
missing values as well as the found outliers.
\end{description}

The macros are executed using the {\tt run\_nda\_command("macro options");}
command.


Both macros expect similar type of input file with a predefined
name ({\tt data}) and they understand the same set of parameters, 
which are provided via sixteen (16) of commandline parameters (global for
all variables) and a special parameter file ({\tt sfr}) that 
specifies variable dependent options.

For example, the  missingness is indicated via a special {\tt missing value},
which is given when calling the macros ({\tt edit.cmd} and {\tt impute.cmd}).
An example of this is included the attached Visual C program.

The NDA/DLL editing and imputation interface requires
that the input data and variable parameters data are set before a call. 

The input data, named as {\tt data}, and the variable specific 
options, named as {\tt sfr}, must be uploaded  to the NDA namespace
before calling editing or imputation routines.

After the operation is executed the new (clean) data will be named
as {\tt output} data frame in the NDA namespace. Similarly the
error probabilities (if used) are named as a {\tt errors} data frame.
The output contains imputed and/or edited records only, and error
probabilities data contain only probabilities for the erroneous
records.
\newpage

To summarize, the steps for the imputation, and editing operations through
the NDA/DLL interface are:

\begin{description}
\item[{\bf Imputation}]:\newline
\begin{enumerate}
        \item create data {\tt data} and attach variables to it,
        \item build the variable spesific options data {\tt sfr},
        \item call macro {\tt impute.cmd} with commandline options, and
        \item read result from a data frame {\tt output}.
\end{enumerate}
%
\item[{\bf  Editing}]:\newline
\begin{enumerate}
        \item create data {\tt data} and attach variables to it,
        \item build the variable spesific options data {\tt sfr},
        \item call macro {\tt edit.cmd} with commandline options, and
        \item read results from a data frame {\tt errors}.
\end{enumerate}
%
\item[{\bf Both editing and imputation steps}]:\newline
\begin{enumerate}
        \item create data {\tt data} and attach variables to it,
        \item build the variable spesific options data {\tt sfr},
        \item call macro {\tt edit.cmd} with commandline options, and
        \item read results from data frames {\tt output} and {\tt errors}.
\end{enumerate}
\end{description}

The implementation of various steps can be examined from the
source code of included Visual C example project.


\subsection{\bf The sfr data: editing and imputing options for data variables}

Before calling the editing and imputation macros, the special data
frame {\tt sfr} that includes variable specific optins 
must be set. The data consists of 4 item string
variables which are:

\begin{enumerate}
        \item row: (flag) the imputation action, which can be\newline
          IMP\_NONE (no imputation), \newline
         IMP\_MEAN  (mean imputation), \newline
         IMP\_URAND (uniform distribution random imputation) \newline
         IMP\_NRAND (gaussian distribution random imputation).\newline
%
        \item row: (flag) the editing action, which can be\newline
           EDIT\_NONE (no edit), \newline
           EDIT\_CONTINUOUS (continuous variable edit)\newline
           EDIT\_CATEGORIAL (categorial variable edit).
%
        \item row: (float) edit cut probability, which 
                    is between 0.0 and 1.0.
        \item row: (float) the sigma1 parameter for robust training. 
         In the case of
        continuous data it is the deviation related influence (factor) 
        of the Huber estimator. Its default value is 3.0. 
        In the case of categorial variables it
        is prior cut probability of the estimator, which must 
       be between 0 and 1.
\end{enumerate}



\subsection{\bf The commandline parameters}

In addition to input data and variable specific options the 
macros require a set of commandline parameters, which are
listed in detail in table \ref{tab:macros_commandline_parameters}.
Parameters 13-15 are required only by the editing. Note also
that an increase of parameter 14 decreases the error probabilities.

\begin{table}[h!]
\begin{tabular}{|l|l|l|l|}
\hline
{\bf Parameter} & {\bf Range} & {\bf Description} & {\bf default}\\
\hline
0  & $>$0          & TS-SOM dimension                             & 2\\
1  & $>$0          & TS-SOM layer                                 &\\
2  & 0,1,2         & TS-SOM topology: 0=lattice, 1=ring, 2=TS-VQ  & 0\\
3  & integer       & missing data value                           &\\
4  & $>$0          & lower limit for cluster record count when    & 5\\
   &               & classifying incomplete records               &\\
5  & $>$0.0        & weighting of neighbours                      & 0.5\\
6  & epsilon       & stopping criteria                            & 0.001\\
7  & $>$0          & maximum number of iterations                 & 20\\
8  & $>$0          & number of corrected layers                   & 3\\
9  & 0,1           & training rule: 0=VQ, 1=spread                & 0\\
10 & 0,1           & use lookup table: 0=no,1=yes                 & 1\\
11 & 0,1           & use fullsearch: 0=no, 1=yes                  & 0\\
12 & 0,1           & use Huber estimator: 0=no,1=yes              &\\
13 & 0,1           & do outlier imputation: 0=no,1=yes            &\\
14 & $>$0.0        & Sigma2, affects to continuous variables'     & 1.0\\
   &               & error probabilities                          &\\
15 & $>$0.0        & Sigma1, training robustness parameter        & 3.0\\
\hline
\end{tabular}
\caption{The commandline parameters for the macros}
\label{tab:macros_commandline_parameters}
\end{table}


\section{NDA/DLL distribution CD.}

\label{sec:nda_files}

This section is the contents of the NDA/DLL distribution.
Table \ref{tab:nda_dll_requires_files} is a list of files required
to compile NDA DLL with other programs (or DLLs).

The DLL functions' return 0 (= OPERATION\_OK, defined in errors.h)
or -10000 (= IGNORE, defined in errors.h) on success. Any other
return value is an error.

\begin{table}[h!]
\begin{tabular}{|l|l|}
\hline
{\bf File} & {\bf Description}\\
\hline
nda.h           & DLL functions' and helper functions declarations\\
ndadll.h        & DLL function type declarations\\
ndadll.cpp      & C interface for the DLL\\
errors.h        & general errors\\
hdr.h           & som errors\\
nda.dll         & the NDA dll (binary)\\
\hline
\end{tabular}
\caption{Required files}
\label{tab:nda_dll_requires_files}
\end{table}

\subsection{\bf Helper functions in ndadll.cpp}

A couple of helper functions have been defined in ndadll.cpp.
Function {\tt init\_nda\_dll} initializes the NDA/DLL interface, it
returns 0 on success, and it must be called at first. Function {\tt
run\_nda\_command} can be called, as other NDA/DLL functions, after
the NDA/DLL interface has been initialized. This function executes a NDA
macro command (or a macro file). Function {\tt free\_nda\_dll}
releases NDA/DLL from memory, and it must be called when
exiting from the program (but note that there are memory leaks in the
current Beta level implemtation).

\subsection{Visual C example project: ndaexample}


\label{sec:vc_example}

The included example Visual C project, ndaexample, is a
demonstration how NDA DLL can be used. The project is a 32-bit
Windows console application. It includes main file {\tt
ndaexample.cpp}, files listed in table
\ref{tab:nda_dll_requires_files}, and files {\tt StdAfx.cpp} and
{\tt StdAfx.h}, which are both generated by Visual C. Two external
NDA macro files, {\tt impute.cmd} and {\tt edit.cmd}, are required.
The files must be placed in the program's working directory.

The function {\tt generate\_data} shows how to create a
new data frame into NDA namespace and how to attach variables to
it. It also creates 5 missing data values and 5 obvious errors in
the data. Function {\tt display\_output} retrieves changed (=
edited or/and imputed) data from NDA's namespace. Edit error
probabilities are read by function {\tt
display\_error\_probabilities}. Function {\tt do\_edit} does error
detection, and it only creates error probabilities, whereas {\tt
do\_imputation} does imputation and creates only imputed data. Edit
and imputation are done by function {\tt do\_edit\_and\_imputation}
which created edited data and error probabilities.

The actual editing is done by running NDA macro {\tt edit.cmd},
whereas {\tt impute.cmd} does imputing.

\subsection{NDA DLL functions}

\label{sec:dll_functions}
Table \ref{tab:exported_dll_functions} contains list of six
necessary DLL functions for data creating, data reading, data
writing, and executing NDA macros. The return value of DLL function
is 0={\tt OPERATION\_OK} or -10000={\tt IGNORE} on success,
otherwise function call failed.

\begin{table}[h!]
\begin{tabular}{|l|}
\hline
{\bf Function and description}\\
\hline
{\tt int nda\_run\_command(char *command, char *ret\_space, long int len)}\\
executes NDA macro {\tt command} and returns output to {\tt ret\_space}.\\
\\
{\tt long nda\_create\_new\_frame(char *framename)}\\
creates a new data frame named {\tt framename}.\\
\\
{\tt long nda\_insert\_ffield(char *frame,char *name, long len,float *vect)}\\
inserts 32-bit float variable to frame {\tt frame} and names the variable as {\tt name}, \\
variable's length is {\tt len} and its data is pointed by pointer {\tt vect}.\\
\\
{\tt long nda\_insert\_ifield(char *frame,char *name, long len,long *vect)}\\
inserts 32-bit signed integer variable to frame {\tt frame} and names the variable \\
as {\tt name}, variable's length is {\tt len} and its data is pointed by pointer {\tt vect}.\\
\\
{\tt long nda\_insert\_sfield(char *frame,char *name, long len,char **vect)}\\
inserts 8-bit string variable to frame {\tt frame} and names the variable as {\tt name}, \\
variable's length is {\tt len} and its data is pointed by pointer {\tt vect}.\\
\\
{\tt long nda\_get\_field\_ptr(char *name, long index, void **ptr,long *nof)}\\
gets data pointer (int*, float* or char**) for {\tt index}:th variable in data frame {\tt name}\\
to pointer {\tt ptr}. The length of data is stored to {\tt nof}.\\
\hline
\end{tabular}
\caption{Exported DLL functions}
\label{tab:exported_dll_functions}
\end{table}

\end{document}
