LuxIO – Installation and Test of the NoSQL key value database

Lux IO is a small performant key value storage. It is B+-tree based and is quite space efficient and fast for large data sets. I tested it on my laptop with 20 million strings.

Installation

cd /tmp
wget http://luxio.sourceforge.net/luxio-0.2.2.tar.gz
tar xvf luxio-0.2.2.tar.gz
cd luxio-0.2.2/
sed -i 's//\n#include /g' util.h
#add line "include " in util.h
#http://ubuntuforums.org/showthread.php?t=346759
./configure
make
sudo make install

#--Add /usr/local/lib path ----------------------
# make installs the library copies to /usr/local/lib
# add /usr/local/lib into the /etc/ld.so.conf file
# Also fixes this error
# Error: libkyotocabinet.so.16: cannot open shared object file: No such file or directory
# More sane default configurations - the default 50% free memory allocation is too much for me
if [ "`sudo grep '/usr/local/lib' /etc/ld.so.conf`" == "" ] ; then
   sudo tee -a  /etc/ld.so.conf <<"_EOF_"
   /usr/local/lib
_EOF_

#sudo ldconfig -v
sudo ldconfig




LuxIO Client Libraries

Java: http://sourceforge.jp/projects/jrcsv/releases?package_id=8658
Perl: http://search.cpan.org/~kentaro/Lux-IO-0.07/lib/Lux/IO.pm
PHP: http://github.com/kajidai/php-luxio
Python: https://github.com/gonsuke/python-luxio#readme
Ruby: http://github.com/kajidai/rluxio/tree/master


Python - Using the python library to LuxIO

Install Cython
sudo apt-get install cython -y

Install python-luxio
cd /tmp
git clone https://github.com/gonsuke/python-luxio.git
cd python-luxio/
sudo python setup.py  install


Test
mkdir ~/build/luxio/python-test
cd /build/luxio/python-test
echo "
import luxio
io = luxio.LuxIO('test')
print io.put('key', 'value')
# True
print io.get('key')
#'value'
print io.append('key', '123')
#True
print io.get('key')
#'value123'
print io.delete('key')
#True
" > luxio-simple-test.py
python luxio-simple-test.py

1 million string puts on my notebook...
mkdir -p ~/build/testluxio
cd ~/build/testluxio

tee ./luxio_perftest.py <<"_EOF_"
import luxio

def main():
    import sys
    #print "argurment count",len(sys.argv)
    if len(sys.argv) < 4:
        print """enter operation and how many.
          example:
          time python luxio_test.py   
          # write 1 million values to testdb
          time python luxio_test.py test.db write 1000000

          # read 1 million values to testdb
          time python luxio_test.py test.db read 1000000
        """
        return
    filename,ops,max = sys.argv[1],sys.argv[2],int(sys.argv[3])
    db = luxio.LuxIO(filename)
    if 'write' in ops:
       mystring='a'*80
       for i in range(0,max):
           db.put('key'+str(i), str(i)+mystring)
    elif 'read' in ops:
       for i in range(0,max):
           db.get('key'+str(i))
    else:
        print 'unkown operation:%s use "read" or "write"'%ops
    print 'done'

if __name__ == "__main__":
    main()

_EOF_

# 1 million write
time python luxio_perftest.py testdb write 1000000
#done
#
#real	0m5.929s
#user	0m2.863s
#sys	0m2.775s

# 1 million reads
time python luxio_perftest.py testdb read 1000000
#done
#
#real	0m2.233s
#user	0m1.624s
#sys	0m0.607s

# 100 million writes
time python luxio_perftest.py bigdb write 100000000
#done
#
#real	9m59.990s
#user	5m7.380s
#sys	4m45.331s


# 100 million reads
time python luxio_perftest.py bigdb read 100000000
# Killed
#
#real	6m18.716s
#user	2m30.402s
#sys	1m16.277s
# memory usage > 70 % of 8GB
# triggered Out Of Memory Killer



time python luxio_perftest.py bigdb read 10000000
# done
#
#real	1m29.855s
#user	0m25.567s
#sys	0m15.247s


Troubleshooting
1. LuxIO Installation Errors

Problem:
make
...
make  all-am
make[1]: Entering directory `/srv/python-luxio/luxio-0.2.2'
if /bin/bash ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I. -I.     -g -O2 -MT btree.lo -MD -MP -MF ".deps/btree.Tpo" -c -o btree.lo btree.cpp; \
	then mv -f ".deps/btree.Tpo" ".deps/btree.Plo"; else rm -f ".deps/btree.Tpo"; exit 1; fi
mkdir .libs
 g++ -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -MT btree.lo -MD -MP -MF .deps/btree.Tpo -c btree.cpp  -fPIC -DPIC -o .libs/btree.o
In file included from btree.cpp:20:0:
util.h: In function 'ssize_t Lux::IO::_read(int, void*, size_t)':
util.h:109:29: error: 'perror' was not declared in this scope
util.h: In function 'ssize_t Lux::IO::_write(int, const void*, size_t)':
util.h:130:30: error: 'perror' was not declared in this scope
util.h: In function 'bool Lux::IO::_pread(int, void*, size_t, off_t)':
util.h:151:29: error: 'perror' was not declared in this scope
util.h: In function 'bool Lux::IO::_pwrite(int, const void*, size_t, off_t)':
util.h:173:30: error: 'perror' was not declared in this scope
util.h: In function 'void* Lux::IO::_mmap(int, size_t, int)':
util.h:195:28: error: 'perror' was not declared in this scope
make[1]: *** [btree.lo] Error 1
make[1]: Leaving directory `/srv/python-luxio/luxio-0.2.2'
make: *** [all] Error 2

Solution:

Add the following line to util.h additional to the other import lines
vi util.h
#include 
#see also http://ubuntuforums.org/showthread.php?t=1554064

Automatically 🙂
sed -i 's//\n#include /g' util.h
#add line "include " in util.h
#see: http://ubuntuforums.org/showthread.php?t=346759

2. Python Library Debugging
Problem:

sudo easy_install python-luxio
...
Downloading http://pypi.python.org/packages/source/p/python-luxio/python-luxio-0.0.3.tar.gz#md5=9c3688c06a05862f8f67420404183f5d
Processing python-luxio-0.0.3.tar.gz
Running python-luxio-0.0.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-tZfMY2/python-luxio-0.0.3/egg-dist-tmp-n3BMAw
Traceback (most recent call last):
  File "/usr/bin/easy_install", line 9, in 
    load_entry_point('distribute==0.6.24dev-r0', 'console_scripts', 'easy_install')()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1931, in main
    with_ei_usage(lambda:
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1912, in with_ei_usage
    return f()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1935, in 
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 368, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 608, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 638, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 828, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1105, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1094, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 30, in run_setup
    lambda: execfile(
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 72, in run
    return func()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 32, in 
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 33, in 
  File "/usr/lib/pymodules/python2.7/Cython/Distutils/extension.py", line 82, in __init__
    **kw)
TypeError: unbound method __init__() must be called with Extension instance as first argument (got Extension instance instead)

Solution:

Manually download the luxio python library from github as above
3. Pip install does not work
sudo pip install python-luxio

fix: install library from github
4.LuxIO Library Import Error

    import luxio
  File "/usr/local/lib/python2.7/dist-packages/luxio/__init__.py", line 1, in 
    from luxio._luxio import *
ImportError: libluxio.so.0: cannot open shared object file: No such file or directory

Solution:
sudo vi /etc/ld.so.conf
# add the following line
/usr/local/lib

sudo ldconfig

References
1. http://luxio.sourceforge.net/ -

   - Is LuxIO faster than Tokyo Cabinet ?

2. http://d.hatena.ne.jp/antipop/20090512/1242126238

   - LuxIO as a storage engine for kyoto Tycoon

3. http://alpha.mixi.co.jp/blog/?p=847

   - CDB

4. http://cr.yp.to/cdb.html