Gugu Ncube

, , , , , ,

beanstalkc – beanstalkd queue – python client library

—

by

Gugu Ncube
in , , , , , ,

Installation

beanstalkc is a c-based python client library for beanstalkd.

Installation using PIP Installer

The library and dependencies can be easily install using the pip installer

sudo pip install PyYAML
sudo pip install beanstalkc

Installation directly from Github

This can useful if the pip installed library is too old and
required new functionality/bug fixes are already in the github source repository.

# create download and build directory
mkdir -p ~/build/beanstalkc
cd ~/build/beanstalkc

# install PyYAML
sudo pip install PyYAML

# download
git clone https://github.com/earl/beanstalkc
cd beanstalkc

# install
sudo python setup.py install

Troubleshooting

1. Error PyYAML

ERROR:root:Failed to load PyYAML, will not parse YAML

Cause: missing PyYAML libraries
Fix: install using pip: sudo pip install PyYAML

2. OUT_OF_MEMORY Exception

    c.put(data)
  File "/usr/local/lib/python2.7/dist-packages/beanstalkc-0.2.0-py2.7.egg/beanstalkc.py", line 120, in put
    ['INSERTED', 'BURIED'])
  File "/usr/local/lib/python2.7/dist-packages/beanstalkc-0.2.0-py2.7.egg/beanstalkc.py", line 95, in _interact_value
    return self._interact(command, expected_ok, expected_err)[0]
  File "/usr/local/lib/python2.7/dist-packages/beanstalkc-0.2.0-py2.7.egg/beanstalkc.py", line 78, in _interact
    raise UnexpectedResponse(command.split()[0], status, results)
beanstalkc.UnexpectedResponse: ('put', 'OUT_OF_MEMORY', [])

Fix: check the setting for the maximum binary log size
In my case, it was: ‘binlog-max-size’: 10485760
=> which is 10485760/(1024*1024)= 10MB

The binlog size is defined by the “-s” startup parameter.
See: man beanstalkd

...
-s:  -s    The maximum size in bytes of each binlog file.
...
sudo /usr/bin/beanstalkd -d -s 10 -b /var/lib/beanstalkd -l 0.0.0.0 -p 11300

The “-s” can also be added as a startup parameter in /etc/default/beanstalkd

Test


cd /tmp

cat > bean_test.py <<- "_EOF_"
import beanstalkc
c = beanstalkc.Connection(host='localhost', port=11300)
print ( c.put('hey!'))
job = c.reserve()
print (job.body)
print(job.delete())
_EOF_

python bean_test.py
#1
#hey!
#None

Mini Benchmark

With 1 Million records ...


cd /tmp

cat > mini-bechmark.py <<- "_EOF_"
# -*- coding: utf-8 -*-
import beanstalkc
import sys, time, datetime

def main():
    # time python test.py
    if len(sys.argv) < 2:
        print("need write or read parameter")
        return
    c = beanstalkc.Connection(host='localhost', port=11300)

    i, action = 0, sys.argv[1]
    print( str(datetime.datetime.now())+" start "+action+"  ... ")
    t1 = time.time()
    if action == 'write':
        i = queue_write(c, 1000000)
    else:
        i = queue_read(c)
    t2= time.time()
    duration = (t2-t1)*1000.0
    text = str(datetime.datetime.now())+" "+action+" "+str(i)+" entries in "+('% 4.f ms' % duration)
    print(text)

def queue_write(c, max):
    for i in range(1,max+1):
        c.put('queue entry!'+str(i))
    return i

def queue_read(c):
    i,job=0,True
    t1 = time.time()
    while job:
        job = c.reserve(timeout=0)
        #print (job.body)
        #print(job.delete())
        if job:
            job.delete()
            i+=1
    return i

if __name__ == '__main__':
    main()

_EOF_

time mini-bechmark.py write
#2013-04-15 11:56:16.491327 start write  ...
#2013-04-15 11:57:45.258481 write 1000000 entries in  88767 ms
#
#real	1m28.837s
#user	0m36.202s
#sys	0m22.773s


time mini-bechmark.py read
#2013-04-15 11:59:07.470119 start read  ...
#2013-04-15 12:02:02.500905 read 1000000 entries in  175031 ms
#
#real	2m55.091s
#user	1m17.993s
#sys	0m45.579s

References

1. beanstalkd home: http://kr.github.io/beanstalkd/
2. beanstalkc pip home: https://pypi.python.org/pypi/beanstalkc
3. beanstalkc tutorial: http://beanstalkc.readthedocs.org/en/latest/tutorial.html
4. beanstalkc reference: https://raw.github.com/kr/beanstalkd/master/doc/protocol.txt