interfacing Python and Torch7(Lua) via shared library

Question

I am trying to pass data (arrays) between python and lua and I want to manipulate the data in lua using the Torch7 framework. I figured this can best be done through C, since python and lua interface with C. Also some advantages are that no data copying is needed this way (passing only pointers) and is fast. I implemented two programs, one where lua is embedded in c and one where python passes data to c. They both work when compiled to executable binaries. However when the c to lua program is instead made to be a shared library things don’t work. The details: I’m using 64-bit ubuntu 14.04 and 12.04. I’m using luajit 2.0.2 with lua 5.1 installed in /usr/local/ Dependency libs are in /usr/local/lib and headers are in /usr/local/include I’m using python 2.7 The code for the c to lua program is: tensor.lua ``` require 'torch' function hi_tensor(t) print(‘Hi from lua') torch.setdefaulttensortype('torch.FloatTensor') print(t) return t*2 end ``` cluaf.h ``` void multiply (float* array, int m…

Accepted Answer

On Linux Lua modules don't link to the Lua library directly but instead expect to find the Lua API functions already loaded. This is usually done by exporting them from the interpreter using the `-Wl,-E` linker flag. This flag only works for symbols in executables, not shared libraries. For shared libraries there exists something similar: the `RTLD_GLOBAL` flag for the `dlopen` function. By default all shared libraries listed on the compiler command line are loaded using `RTLD_LOCAL` instead, but fortunately Linux reuses already opened library handles. So you can either:

Preload the Lua(JIT) library using `RTLD_GLOBAL` before it gets loaded automatically (which happens when you load `libcluaf.so`):

```
from ctypes import byref, cdll, c_int
import ctypes

lualib = ctypes.CDLL("libluajit-5.1.so", mode=ctypes.RTLD_GLOBAL)
l = cdll.LoadLibrary('absolute_path_to_so/libcluaf.so')
# ...
```

Or change the flags of the Lua(JIT) library handle afterwards using the `RTLD_NOLOAD` flag for `dlopen`. This flag is not in POSIX though, and you probably have to use C to do so. See e.g. here.

interfacing Python and Torch7(Lua) via shared library

Solution

Problem