Matlab MEX function

[smorosohk]

Call the EPED FANN and TENSORFLOW neural nets from MATLAB

[orso82]

@smorosohk I see that you have re-coded the c library in your matlab MEX functions. Please see if you can just link to the libbrainfusetf.a and libbrainfuse.a that the make libs command generate.

Concerning the issue that you are seeing, getting 10 outputs instead of the 18 outputs that you expect from EPED1-NN... my suspicion is that somehow you are mixing up inputs (of which there are 10) and outputs. Please check if that's the case somewhere.

[smorosohk]

I did originally attempt to link brainfusetf_lib to the MEX function, and was getting errors that it did not link correctly. It was simpler to copy the function definitions into the MEX function.

I don't believe that I am mixing up inputs and outputs. I am using 10 inputs, and the 10 outputs that I am getting are the same values as the first 10 outputs when I run brainfusetf_run.exe from the terminal. I have also hardcoded 18 as the number of outputs I should be getting from the btf_run function (Call_EPED_NN.c line 248).

[orso82]

@smorosohk can you please try to change the number of tries that are done at this line: https://github.com/gafusion/neural/blob/master/brainfusetf_lib.c#L133
from 10, to say 100

Any difference?

[smorosohk]

When I run that for loop more than 10 times it causes MATLAB to crash. I get the "error on read" error message from ReadXBytes and I am forced to restart MATLAB.

[smorosohk]

It looks like I was initializing the output array with a size of 10. I am now getting an output vector of the correct length, but MATLAB is still crashing on this line:

neural/brainfusetf_lib.c

Line 158 in e5fc175

snprintf(message2,strlen(pToken[2]),",%s",pToken[2]+1);

I'm guessing the issue is coming from pToken, which is a constant 1.4429 (equal to the last input). Is message1 not being parsed correctly in line 156?

I am also still getting error on read from this line:

neural/brainfusetf_lib.c

Line 145 in e5fc175

ack+=ReadXBytes(sockfd, sizeof(length), (void*)(&length));

[smorosohk]

I figured out how to get all of the outputs and not have MATLAB crash. The version in the matlab -1 branch is calling EPED correctly from MATLAB. Can you tell me where to find the .pb file for the TGLF model so I can work on calling that from MATLAB?

[orso82]

@smorosohk with b0d7f8e I have added the TGLF models. You want to use the 2IONS model for DIII-D.

[smorosohk]

Thank you Orso. Does the 2IONS model have the same 23 inputs and 4 outputs as the model in your paper?

[orso82]

I think it has 23 inputs and 5 outputs (I added the carbon particle flux since the paper). Use the info function to see what these are.

[smorosohk]

I have not actually been able to get the .info() function to work. As far as I can tell it is not a C function so it does not work in my MEX function. It is a python function, but I'm not actually running anything in python and when I try to run it from the terminal it does not work.

[orso82]

Can you try running the tf_client_server_test.py script? If you get an error, what error do you get?

[smorosohk]

I get a permission denied error.

[orso82]

how are you running the script? Are you doing python tf_client_server_test.py ?

[smorosohk]

I was running python first and then the script. If I run them in the same line I get "No module named SocketServer"

[orso82]

This looks like to be an issue with the python that you have on the system that you are running.

Do you have access to the iris cluster at GA?
Can you login and do:

module load omfit
cd DIRECTORY_WHERE_YOU_HAVE_NEURAL_INSTALLED
python tf_client_server_test.py

[smorosohk]

That worked from iris, thank you. I'm not sure what the issue was on my computer, but if I can access the info on iris that's all I need.

[orso82]

👍

[smorosohk]

I have both the EPED and TGLF neural nets working from MATLAB, and have uploaded them to the matlab-1 branch.

[smithsp]

@smorosohk
I am glad that you got it working. I deleted the binary and backup files that you had uploaded to this branch.
@orso82 This branch should be squashed to remove any trace of the binary files.

[orso82]

This is excellent progress, but I still have an issue with the brainfuse library being copied an pasted in the matlab function source code. @smorosohk can you try one more time to avoid doing that. I think all you need to do is pass the path to the libbrainfusetf.a file when you compile your MEX function.

[smithsp]

@orso82 It may need a dynamic library instead of a static one.

[orso82]

@smorosohk I have updated the Makefile to generate dynamic libraries. If you do a git pull and run a make all you should get .so files in the neural folder. You may then be able to remove the redundant code in your MEX function.

[smorosohk]

I haven't been able to set up the fann library on my personal computer, so when I try to run make all I get an error message that the file 'fann.h' was not found. I tried to set it up the same way I did on my office computer before that computer died, which worked, but for some reason it does not work on my personal computer.

[orso82]

Are you using the FANN models at all?

[smorosohk]

I was, I got the EPED FANN model to work in MATLAB before my office computer died. Having to write the inputs to a file takes longer to run, so I've been focusing on the TENSORFLOW models.

[orso82]

If you are not using the FANN models, than you should be able to generate the dynamic library for the tensorflow models by running make libbrainfusetf.so

[smorosohk]

I was able to make the libbrainfusetf files, but I have still not been able to link them to the MEX functions. I have tried every way I can think of -both in the source code and when compiling- to link to libbrainfusetf.a, and the MEX function stops even attempting to compile after it hits 20 errors. I know that copying all of the function definitions is not the best coding practice, but it is the only way I can see to make the MEX functions compile.

[smithsp]

@smorosohk Can you post the errors you are encountering when trying to link the lib files to the MEX files?

[smorosohk]

I am getting over 400 warnings, literally on every character of libbrainfusetf.a, most of which are "warning: null character ignored [-Wnull-character]". Most of the 20 errors are "error: source file is not valid UTF-8"

[smithsp]

What is the command to generate the mex library?

[smorosohk]

mex Call_EPED_NN.c

I have tried linking libbrainfusetf.a in the generation command using
mex -I"libbrainfusetf.a" Call_EPED_NN.c
and as many variations of that as I can come up with or find in MEX tutorials.

[smithsp]

Based on a standard compiler/linker, I would have guessed

mex -lbrainfusetf Call_EPED_NN.c

What does that return?

[smorosohk]

I get the same errors.