Wednesday, January 29, 2014

Python? , But why Python?

Well I am using python since last 3 years . It has been my main development language, other than javascript.

Often, someone comes around and asks , Why Python? I am usually left with my personal programming tasks where python comes in very handy when compared to other languages I have encountered, like  PHP or Java.

Well , here is a post for answering this question exactly. Check it here.

Here are the points -

  • Efficient - Has generators 
  • Fast
  • Broad use
  • Not just a language, has a number of implementations
  • Easy

Cheers!!

Yet another post on Redis

While working for a project , we used Redis as queue, using python-rq.  Running a redis-cli , I used the following commands -


  • keys *
  • type <key name>
  • and then according to the type , hash,list I would query the data
Some things were quite easy to understand
  • rq:workers
  • rq:queue:failed
  • rq:queue:default
  • and a success one as well
But apart from these, there were several entries - with name rq:job:<job_id>. After much reading, I found the internal working at http://python-rq.org/contrib/.

It says whenever a function call gets enqueued - 
  • Pushes the job's ids into queue , in my case the default
  • adds a hash objects of the job instance
So, when dequeue happens - 
  • Pops jobid from queue
  • Fetches Job data 
  • Executes the function and saves result has a hash key if success
  • else saves in failed queue with stack trace
All of this is given on Python-rq site.

There are two kinds of error I saw -
  • RuntimeError: maximum recursion depth exceeded while calling a Python object - This happened at queue.py of python-rq module, where I think, it was caused when control crossed max recursive limit, when it didnt find the jobs hashes, as discussed above in dequeue
  • Socket closed on remote end - The server closes client connection after 300s, in my case I didnt want to do them, so. let it be on forever by changing in /etc/redis/redis.conf , timeout value to 0
Go Redis!!

Sunday, January 19, 2014

Python Decorators - The correct way to do it

Was going through Graham Dumpleton's blog post - how you implemented your python decorator is wrong. Simple points that were discussed were -


  • Decorators can be functions as well as Classes.
    As a class -  

    class function_wrapper(object):
        def __init__(self, wrapped):
            self.wrapped = wrapped
        def __call__(self, *args, **kwargs):
            return self.wrapped(*args, **kwargs) 
    @function_wrapper
    def function():
        pass 

    As a function 

    def function_wrapper(wrapped):
        def _wrapper(*args, **kwargs):
            return wrapped(*args, **kwargs)
        return _wrapper 
    @function_wrapper
    def function():
        pass 

  • Use the functools.wraps decorator , it only preserves original functions __name__ and __class__
    In functions 

    import functools 
    def function_wrapper(wrapped):
        @functools.wraps(wrapped)
        def _wrapper(*args, **kwargs):
            return wrapped(*args, **kwargs)
        return _wrapper 
    @function_wrapper
    def function():
        pass 

    In classes, use the update_wrapper method- 

    import functools 
    class function_wrapper(object):
        def __init__(self, wrapped):
            self.wrapped = wrapped
            functools.update_wrapper(self, wrapped)
        def __call__(self, *args, **kwargs):
            return self.wrapped(*args, **kwargs)
  • Python 2.7 preserves Argument specification ( inspect.getargspec ) only in functional decorators, not in class based ones.
  • Doesnt preserve function source code for inspection ( inspect.getsource ) 
  • Cannot apply decorators on top of other decorators that are implemented as descriptors.

Saturday, January 11, 2014

Failed to attach to key daemon - Error in Shrew Soft VPN

Using Shrew Soft VPN on Ubuntu 12.04, I often face this error

Failed to attach to key daemon 

On some googling I found this post -> Ubuntu Forums

It says that this error is because IKE Daemon isnt running.

sudo iked

Hope this helps.

Friday, December 20, 2013

Solving failed to load canberra-gtk-module error on precise.

I have Ubuntu Precise(12.04) 64 bit.

On openinig sublime, it gave me this error -
(sublime:7247): Gtk-WARNING **: Unable to locate theme engine in module_path: "murrine",
(sublime:7247): Gtk-WARNING **: Unable to locate theme engine in module_path: "murrine",
(sublime:7247): Gtk-WARNING **: Unable to locate theme engine in module_path: "murrine",
Gtk-Message: Failed to load module "canberra-gtk-module"
'import site' failed; use -v for traceback
`menu_proxy_module_load': sublime: undefined symbol: menu_proxy_module_load
(sublime:7247): Gtk-WARNING **: Failed to load type module: (null)


on running strace on sublime ,

I got this output  -

access("/usr/lib/i386-linux-gnu/gtk-2.0/modules/libcanberra-gtk-module.la", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/2.10.0/i686-pc-linux-gnu/modules/libcanberra-gtk-module.so", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/2.10.0/i686-pc-linux-gnu/modules/libcanberra-gtk-module.la", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/2.10.0/modules/libcanberra-gtk-module.so", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/2.10.0/modules/libcanberra-gtk-module.la", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/i686-pc-linux-gnu/modules/libcanberra-gtk-module.so", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/i686-pc-linux-gnu/modules/libcanberra-gtk-module.la", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/modules/libcanberra-gtk-module.so", F_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/gtk-2.0/modules/libcanberra-gtk-module.la", F_OK) = -1 ENOENT (No such file or directory)
stat64("libcanberra-gtk-module.so", {st_mode=S_IFREG|0644, st_size=23072, ...}) = 0
open("/lib/i386-linux-gnu/libcanberra-gtk-module.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/i386-linux-gnu/sse2/libcanberra-gtk-module.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/i386-linux-gnu/libcanberra-gtk-module.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libcanberra-gtk-module.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libcanberra-gtk-module.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "Gtk-Message: Failed to load modu"..., 57Gtk-Message: Failed to load module "canberra-gtk-module"

 As the error says, it was trying to locate libcanberra-gtk-module.so in a number of locations.

Googling the package name , I saw that the location in which it was trying to find was wrong, in locations where 32bit packages are there.

There was the solution. I was trying to load 32 bit Sublime Text on 64 bit system. Downloading a 64 bit version worked like a charm.



Tuesday, October 1, 2013

Ejabberd Dissection

Working on Ejabbered 2.1.13, in order to understand ejabbered db calls and processes, I followed this presentation  - http://www.slideshare.net/ShaojieYang/ejabberd-installation-configuration-15508995
.
 It showed an easy process to connect Ejabberd to an external database.
Here I would be going a step ahead and saving the DB calls Ejabberd makes, assuming this will be same, when it using its internal Mnesia DB.

When adding a new user(registration)  from web admin - test3@192.168.1.250
----------------------------------------------------------------------------------------------------------
  113 Query select password from users where username='admin'
  112 Query select password from users where username='admin'
  112 Query select password from users where username='test3'

  104 Query insert into users(username, password) values ('test3', 'test3')

Below for count for Offline Messages and  Last Activity shown in admin
-----------------------------------------------------------------------
108 Query select username from users
104 Query select count(*) from spool  where username='admin'
104 Query select seconds, state from last where username='admin'
107 Query select count(*) from spool  where username='admin3'
108 Query select seconds, state from last where username='admin3'

Connecting a user - test2@192.168.1.250
-------------------------------------------------------
109 Query select password from users where username='test2'
113 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test2'
104 Query select name from privacy_default_list where username='test2'
105 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test2'
112 Query select jid, grp from rostergroups where username='test2'
109 Query select password from users where username='test2'
113 Query select name from privacy_default_list where username='test2'
105 Query select name from privacy_list where username='test2'
105 Query select id from privacy_list where username='test2' and name='invisible'
110 Query begin
110 Query select id from privacy_list where username='test2' and name='invisible'
110 Query insert into privacy_list(username, name) values ('test2', 'invisible')
110 Query select id from privacy_list where username='test2' and name='invisible'
110 Query delete from privacy_list_data where id='1'
110 Query insert into privacy_list_data(id, t, value, action, ord, match_all, match_iq, match_message, match_presence_in, match_presence_out ) values ('1', 'n', '', 'd', '1', '0', '0', '0', '0', '1')
110 Query commit
104 Query begin
104 Query select username, xml from spool where username='test2'  order by seq
104 Query delete from spool where username='test2'
104 Query commit
108 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test2'
110 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test2'
113 Query select jid, grp from rostergroups where username='test2'
105 Query select subscription from rosterusers where username='test2' and jid='test2@192.168.1.250'
112 Query select subscription from rosterusers where username='test2' and jid='test2@192.168.1.250'
112 Query select subscription from rosterusers where username='test2' and jid='test2@192.168.1.250'
104 Query select subscription from rosterusers where username='test2' and jid='test2@192.168.1.250'

111 Query select subscription from rosterusers where username='test2' and jid='test2@192.168.1.250'

Adding contact - test1@192.168.1.250 by test4
--------------------------------------------------------------

122 Query begin
122 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test4' and jid='test1@192.168.1.250'
122 Query update rosterusers set username='test4', jid='test1@192.168.1.250', nick='', subscription='N', ask='N', askmessage='', server='N', subscribe='', type='item' where username='test4' and jid='test1@192.168.1.250'
122 Query insert into rosterusers(username, jid, nick, subscription, ask, askmessage, server, subscribe, type) values ('test4', 'test1@192.168.1.250', '', 'N', 'N', '', 'N', '', 'item')
122 Query delete from rostergroups       where username='test4'         and jid='test1@192.168.1.250'
122 Query commit
131 Query begin
131 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test4' and jid='test1@192.168.1.250'
131 Query select grp from rostergroups where username='test4' and jid='test1@192.168.1.250'
131 Query update rosterusers set username='test4', jid='test1@192.168.1.250', nick='', subscription='N', ask='O', askmessage='', server='N', subscribe='', type='item' where username='test4' and jid='test1@192.168.1.250'
131 Query commit
129 Query select name from privacy_default_list where username='test1'
128 Query select password from users where username='test1'
123 Query begin
123 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test1' and jid='test4@192.168.1.250'
123 Query update rosterusers set username='test1', jid='test4@192.168.1.250', nick='', subscription='N', ask='I', askmessage='', server='N', subscribe='', type='item' where username='test1' and jid='test4@192.168.1.250'
123 Query insert into rosterusers(username, jid, nick, subscription, ask, askmessage, server, subscribe, type) values ('test1', 'test4@192.168.1.250', '', 'N', 'I', '', 'N', '', 'item')
123 Query commit

Now asks for permission 

in rosterusers table - subscription field is N (related to pending as others which are already accepted contacts have B) and  ask field is O for test4 and I for test1 (test4 added test1)


Ater accepting
-------------------

122 Query begin
122 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test4' and jid='test1@192.168.1.250'
122 Query update rosterusers set username='test4', jid='test1@192.168.1.250', nick='', subscription='N', ask='N', askmessage='', server='N', subscribe='', type='item' where username='test4' and jid='test1@192.168.1.250'
122 Query insert into rosterusers(username, jid, nick, subscription, ask, askmessage, server, subscribe, type) values ('test4', 'test1@192.168.1.250', '', 'N', 'N', '', 'N', '', 'item')
122 Query delete from rostergroups       where username='test4'         and jid='test1@192.168.1.250'
122 Query commit
131 Query begin
131 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test4' and jid='test1@192.168.1.250'
131 Query select grp from rostergroups where username='test4' and jid='test1@192.168.1.250'
131 Query update rosterusers set username='test4', jid='test1@192.168.1.250', nick='', subscription='N', ask='O', askmessage='', server='N', subscribe='', type='item' where username='test4' and jid='test1@192.168.1.250'
131 Query commit
129 Query select name from privacy_default_list where username='test1'
128 Query select password from users where username='test1'
123 Query begin
123 Query select username, jid, nick, subscription, ask, askmessage, server, subscribe, type from rosterusers where username='test1' and jid='test4@192.168.1.250'
123 Query update rosterusers set username='test1', jid='test4@192.168.1.250', nick='', subscription='N', ask='I', askmessage='', server='N', subscribe='', type='item' where username='test1' and jid='test4@192.168.1.250'
123 Query insert into rosterusers(username, jid, nick, subscription, ask, askmessage, server, subscribe, type) values ('test1', 'test4@192.168.1.250', '', 'N', 'I', '', 'N', '', 'item')
123 Query commit

Memory footprint 
-----------------------

connecting 100 clients to ejabberd - no issue,
there are 9 instances running with around 70mb each.

Used this script to hit to server (Uses XMPPY )
-----------------------------------------

It assumes that we have test users already registered with username test1, test2.. and pass test1 and test2.  (ejabberdctl register command can be used to make such users)

from xmpp import *

clients = []
for i in range(100):
    clients.append(Client('192.168.1.250'))
for cl in clients:
    cl.connect(server=('192.168.1.250',5222))
for i in range(1,101):
    clients[i-1].auth('test%d'%i,'test%d'%i)

while True:
    for cl in clients:
        cl.sendPresence()

        cl.Process(2)



Sum good post
http://metajack.im/2008/08/27/migrating-to-ejabberd-the-gory-details/



Wednesday, September 11, 2013

Timezone and Confusion in Django

https://docs.djangoproject.com/en/dev/topics/i18n/timezones/

1) We use USE_TZ = True in our settings. It means, that Django will use Time zone aware datetime objects.
so, instead of doing this,


import datetimecur = datetime.datetime.now() # which doesnt give us tzinfo

we should use this -

import datetimefrom django.utils.timezone import utc
cur = datetime.datetime.now().replace(tzinfo=utc) # Note: This is for timezone aware not for local time

This will remove the warnings we usually see, "Datetimefield received a naive datetime."

2) For converting local time into current time zone,  according to one specified in settings file. We should use this

from django.utils import timezonetimezone.localtime(timezone.now())