Beyond bits and bytes

NetMos 9845 Multiserial on Ubuntu 10.04

2012-01-26T09:56:00.002+02:00

I have installed a NetMos 9845 Multiserial (8 port) card on my pc as i need several serial ports for a project.
Problem is Ubuntu does not recognize more than 4 ports by default.
To solve this you need to add a kernel boot parameter to instruct the kernel during boot to look for more serial devices.

With Grub2 on the machine, you can no longer edit the grub kernel parameters directly as Grub2 uses a templeting system.
So use your favorite editor and open (as root)

/etc/default/grub

there you will see a line

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

change it to

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash 8250.nr_uarts=8"

save the file and then issue the command

sudo update-grub

for the changes to take effect

Reboot your machine and check dmesg for the number of serial ports you now have
In my case it looks like this

:/etc/grub.d$ dmesg | grep ttyS
[ 0.709951] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.710235] 00:07: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.710400] 0000:05:01.0: ttyS4 at I/O 0xc000 (irq = 19) is a 16550A
[ 0.710477] 0000:05:01.0: ttyS5 at I/O 0xc100 (irq = 19) is a 16550A
[ 0.710551] 0000:05:01.0: ttyS6 at I/O 0xc200 (irq = 19) is a 16550A
[ 0.710626] 0000:05:01.0: ttyS7 at I/O 0xc300 (irq = 19) is a 16550A
[ 0.710700] 0000:05:01.0: ttyS1 at I/O 0xc400 (irq = 19) is a 16550A
[ 0.710775] 0000:05:01.0: ttyS2 at I/O 0xc500 (irq = 19) is a 16550A

You can also check the number and setting of the serial ports using the setserial command for every port

sudo setserial /dev/ttyS7 -a
/dev/ttyS7, Line 7, UART: 16550A, Port: 0xc300, IRQ: 19
Baud_base: 115200, close_delay: 50, divisor: 0
closing_wait: 3000
Flags: spd_normal skip_test

Getting your Android phone recognized by adb for debugging

2012-01-22T10:36:00.002+02:00

In an older post i have shown hot to get a Samsung phone "recognized" by adb for debugging.
These were the "early days" of Android development...
Since then Android got more mature (so did adb) and now there are many companies producing Android compatible phones that developers want to use for testing/debugging their apps.
One of the main issues people have in Ubuntu is how to get the correct udev rules for their phone, as there are many sites ans post with old info or targeting specific phones.
The other problem developers face is how to use the phone in debug mode without having to run adb as root.

First thing to do is have a go at the official Android dev page
It has (most of) the instruction you need to get your phone connected and recognized.

It all boils down to creating a new udev rule file and adding something like this

SUBSYSTEM=="usb", ATTR{idVendor}=="0bb4", MODE="0666", GROUP="plugdev"

Basically you have to tell udev that devices with idVendor=="0bb4" should get read/write privileges (the MODE="0666" part) for the group "plugdev"
In this example, the vendor ID 0bb4 is for HTC. The MODE assignment specifies read/write permissions, and GROUP defines which Unix group owns the device node.

It also has an extensive list of vendor id's you can use to match your device.

In case your device manufacturer is not in that list, there is an easy way to find the vendor id of your device.
Connect your device and issue

lsusb

and you will get a list of connected usb devices and their associated vendor and device id's

in my case it looks like this

Bus 008 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 015: ID 12d1:1038 Huawei Technologies Co., Ltd.
Bus 001 Device 005: ID 0d8c:000c C-Media Electronics, Inc. Audio Adapter
Bus 001 Device 004: ID 05e3:0702 Genesys Logic, Inc. USB 2.0 IDE Adapter
Bus 001 Device 003: ID 0409:0050 NEC Corp.
Bus 001 Device 002: ID 03f0:4105 Hewlett-Packard ScanJet 4370
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Locate your device (in this case it's the Huawei Technologies Co., Ltd) and the first part of the ID 12d1:1038 (i.e the 12d1) is the idVendor you need you need to put in the udev rules

Another common problem developers face is that although the udev rules are correct the phone does not connect to adb and shows up as a number of ???????????? when issuing an "adb devices" command.

The only way to get the phone connected is to start the adb with root privileges using sudo ,which in my book is a big "NoNo"
To solve this you need to set the correct user and group in the udev rules
Here is how my udev entry looks like

SUBSYSTEM=="usb", SYSFS{idVendor}=="22b8", OWNER="your-user-name" GROUP="your-group"
SUBSYSTEM=="usb", ATTR{idVendor}=="22b8", MODE="0666"

To get your username and group issue the command

id

and look for the uid and gid entries
in my case they look like this

uid=1000(stelios) gid=1000(stelios)

Use the uid name in parenthesis in the OWNER and gid name in the GROUP, restart udev and now you will be able to access the phone from adb as a normal user without having to issue sudo

Running out of inodes

2011-06-19T22:47:00.003+03:00

Here is the problem.
Had a gazillion of small files stored in a 1TB partition and everything was fine, then we need it to move these files to a smaller partition and guess what, we could not.
What happened is that we would run out of inodes in the new smaller partition when copying the data over.

Lesson learned ?
If you create a small partition and have a ton of small files you will run out of inodes, and if you use Ext4 and have flex_bg in features, then you can't use tune2fs to increase inodes in that partition.
So then the only option is to reformat the partition and increase the number of inodes

Some would argue that you could use an other filesystem with dynamic inode allocation, but personally I would go with ext4 over any version of riserfs,xfs whatever as its way more mature.

To avoid the problem do the following when formatting:

mkfs.ext4 -I 512 /dev/foo
tune2fs -i0 -c0 -o journal_data_writeback /dev/foo

Formatting this way will give you way more inodes than a "normal" format would as it increases the inode size from the default 256.

Canon CLC 3200 Ubuntu and Debian Drivers

2011-01-14T11:39:00.004+02:00

If you are looking for the CUPS drivers of the Canon CLC 3200 for Ubuntu and Debian you can get them from here for both 32 and 64 bit version.

http://software.canon-europe.com/products/0010073.asp

Have tested them on Ununtu 10.04 both on 32bit and 64bit machines and works great.

Using the web2py framework on Eclipse

2010-10-04T10:54:00.002+03:00

Here is a nice post on getting web2py to work through Eclipse for faster/better debugging

web2py is one of the best Python frameworks out there are the moment and if you are into Python or looking for a framework to develop the next "killer" web app have a look at it.

Fix asterisk addons problems for Ubuntu

2010-05-27T12:41:00.003+03:00

There is a bug in the recent asterisk-addons package shipped with Ubuntu 10.04
The result is that all the modules that are part of the package fail to load with a message like this

WARNING[13478]: loader.c:800 load_resource: Module 'app_addon_sql_mysql' could not be loaded.

Till this get fixed by the Ubuntu people here a quick fix

sudo aptitude install asterisk-dev
wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-addons-1.6.2.1.tar.gz
tar xvzf asterisk-addons-1.6.2.1.tar.gz
cd asterisk-addons-1.6.2.1
./configure
make

sudo su

cp -a apps/*.so /usr/lib/asterisk/modules/
cp -a cdr/*.so /usr/lib/asterisk/modules/
cp -a channels/*.so /usr/lib/asterisk/modules/
cp -a formats/*.so /usr/lib/asterisk/modules/

Generate passwords with crypt and random salt

2010-05-14T08:36:00.002+03:00

Here is a small perl script i found sometime ago to generate crypted passwords with random salt values

#!/usr/bin/perl

$plain=$ARGV[0]; ## Read the command line argument

if (!$plain){ ## No parameter pased
print "Usage: $0 plain_text_password\n";
exit;
}

## Use the Process id & time to generate the salt.
srand($$|time); # random seed
@saltchars=(a..z,A..Z,0..9,'.','/'); # valid salt chars
$salt=$saltchars[int(rand($#saltchars+1))]; # first random salt char
$salt.=$saltchars[int(rand($#saltchars+1))]; # second random salt char
$newuser = crypt ($newuser, $salt);

## Generate the encrypted password
$crypted=crypt ($plain, $salt);

print $crypted,"\n";

Τα μαύρα χάλια μας και καλό κουράγιο

2010-03-02T08:58:00.003+02:00

Παρακολουθώ τους τελευταίους μήνες όλα αυτά που συμβαίνουν γύρω από τα οικονομικά της χώρας και πραγματικά αναρωτιέμαι ο 2 χρόνος γιός μου σε τι κατάσταση θα ζήσει.

Ξεκινήσαμε από το "η κρίση εμάς δεν πιάνει", ακούσα για λεφτά που υπήρχαν και μετά "εξαφανίστηκαν" και έγιναν κοψίματα μισθών, νέους έμμεσους φόρους (τη χειρότερη μορφή φορολόγησης) μέτρα εδώ, μέτρα εκεί, κομπίνες, ρεμούλες και κατασπατάληση χρημάτων αλλά πουθένα δεν άκουσα κάποιον από τους πολιτικούς μας να αναλαμβάνει ευθύνη για αυτά που έχουν γίνει στην χώρα τα τελευταία 40 χρόνια και μας έχουν οδηγήση σε αυτή τη θέση.

Λέω μήπως, μήπως οι κύριοι βουλευτές, υπουργοί και λοιποί "παρά το ...." (συμπληρώστε το κενό με ότι καρεκλοκένταυρο θέλετε) θα έπρεπε πρώτα να κόψουν τις δικές τους *πραγματικές* αμοιβές 50% και μετά να ζητήσουν από όλους τους υπόλοιπους να "θυσιασούν" μισθούς και λοιπά άλλα ?
Όχι ότι θα σωθούμε έτσι, (γιατί και του χρόνου, αν δεν υπάρξει διακανονισμός τους χρέους, σεισάχθια το είπαν οι αρχαίοι τα ίδια και περισσότερα θα χρωστάμε) άλλα το να βάζουν όλοι αυτοί που έριξαν τη χώρα στα "βράχια" με τις πολιτκές τους,τις αποφάσεις τους και τις λοπές ρεμούλες τα τελευταία 40 χρόνια, τους "άλλους" να την ξελασπώσουν *πάλι*, και αυτοί να κάνουν εξεταστικές και να τσεπώνουν τα 300άρια άνα συνδρίαση είναι λίγο...γυφτιά.
Από την άλλη όμως, οι γύφτοι μια χαρά άνθρωποί είναι και δεν φταίνε σε τίποτα για την σημερινή κατάσταση, οπότε μάλλον θα πρέπει να αλλάξουμε τη λέξη από γυφτιά σε "βουλεφτιά" ή "πολιτεφτιά" γιατί μόνο έτσι μπορούμε να καταλάβουμε το πραγματικό νόημα του τι συμβαίνει.

Και για να μην παρεξηγούμαστε, όχι ότι είμαι αντίθετος στη μείωση της σπατάλης στο Δημόσιο, το κόψιμο των "επιδομάτων" στους αργόσχολους του Δημοσίου (έδω καταντήσαμε να παίρνουν επίδομα για να μην παίρνουν 'φακελάκι'και καλά) από τους οποίους οι μισοί τουλάχιστον θα έπρεπε να πάρουν πόδι και να πάνε σπίτια τους.

Δυστηχώς το μόνο που θα καταφέρουν οι πολιτικοί μας θα είναι μια τρύπα στο νερό, τα νούμερα δεν κάνουν λάθος.
Αν χρωστάς παραπάνω από αυτά που παράγεις δεν υπάρχει περίπτωση ποτέ να μπορέσεις να μειώσεις το χρέος.
Στην καλύτερη περίπτωση μπορείς να το μετατοπίσεις για "λίγο αργότερα" (όπως γίνονταν συστηματικά μέχρι τώρα) αλλά το τέλος θα είναι πάντα το ίδιο. Θα πτωχεύσης.

Είμαστε σαν το ασθενή που είναι στην εντατική και οι "συγγενείς και φίλοι" τον κρατάνε στη ζωή για να μπορέσει να τους γράψει το "οικοπεδάκι" και να πάρουν ότι μπρορούν πριν το "μοιραίο", και το "μοιραίο" ίσως είναι πιο κοντά από όσο εμείς νομίζουμε.

Οπότε, καλό κουράγιο Έλληνες.

Creating OpenOffice Horizontal Lines

2010-02-16T16:20:00.004+02:00

Following is a 'shortcut' for OpenOffice to create horizontal lines

Horizontal lines can be created in Writer by typing a specific symbol three times on a line by itself and then pressing Enter.
The choice of symbol * - = # ~ _ decides the type of line that is created.

You can find more shortcuts and other OpenOfice related stuff here

Solving fax issues in Asterisk

2010-02-08T18:57:00.006+02:00

"Asterisk","fax" and "problem" are three words you find often in forums and threads of people looking for help.
In the age of email,and pdf there are still large number of companies relying in fax to transmit documents.
The reasons are many, but the problems implementors face are common.

Bad quality of faxes with "missing" or malformed lines, large number of failed faxes etc.
Most of this issues come from two facts.
The first is that we are trying to use a medium that was primarily designed for human voice to carry high speed analog modem signals and second,that fax standards are not so "standard".

Putting asterisk to handle faxes can be both a blessing and a curse for those two reasons.

Asterisk has the ability to detect if an incoming (or outgoing) call is a fax and do a number of things, depending if its acting as a "pass-through" or receive the fax.

The first thing Asterisk does is try to detect if the call is a fax call and shutdown the echo canceler on that channel.
For this to work, asterisk listens for the CNG tone emitted by the calling fax and if found, disables the echo canceler in zaptel.

The CNG is in the CCITT (ITU-T) Recommendation T.30 and describes how FAX calls are established.

During the call setup/establishment process, there are two tone signals that are send:

CNG and CED

The CNG signal *may* be sent from the Originating FAX machine after dialing is complete.
The CNG signal consists of the transmission of 1100 Hz for 1/2 second, followed by a 3 second silent (2100 Hz OFF) period.

The CED signal *may* be sent by the Terminating FAX machine anywhere between 1.8 to 2.5 seconds AFTER answering the call. The CED signal consists of a 2100 Hz tone that is from 2.6 to 4 seconds in duration. The CED tone is useful for disabling any echo cancellers on the line.

(notice the *may*, don't you love clearly defined standards ;)

Asterisk then tries to redirect the call flow to the 'fax' extension, if its present in the context that is currently executing in the dialplan.
There you can use an asterisk application to receive faxes or redirect to another port etc.

Recently we faced a problem with one of our clients that is using an E1 for voice and fax calls.
The calls come in through the E1 and the faxes are connected to an 8 port analog card
Some of the faxes received had quality issues and we tried to figure out what the problem was.

After a lot of head scratching and countless tries we realized two things.
The PRI card was somehow "loosing" some frames and that the echo canceler, OSLEC in this case, was not shutting down when a fax was received.
The first issue was easy to solve, as we discovered that the PRI card was sharing the same IRQ as the network card, so switching the pci slot and making sure that the PRI did not share IRQ's with any other device.

The tricky part was to figure out why the echo canceler was not shutting down when the call was a fax call.
The problem with an E1 (or T1) is that the calls don't use "fixed" channel numbers as in a pstn card for example, where you know that number X is on channel Y.
The provider is sending the call to the first available channel and its up to the dialplan to determine where the call would be routed.
Shutting down the echo canceler manually for all of the E1 channels, solved the issue of the faxes but created 'random' problems of echo to the voice channels.

So i started looking in detail (i.e the code) how asterisk detects faxes.

What i realized is that asterisk needs a 3-5 seconds *after* the call is answered to determine if this is a fax call,shutdown the echo canceler and then jump to the 'fax' priority.
If you "bridge" the call to another zap channel (or to sip channel of an ATA for example) before that, then the EC stays on.
And this was exactly what we were doing in our case.
As soon as the call come in, we bridged it to the analog card, not giving enough time to asterisk to shutdown the EC.

So the problem can be solved by adding a delay of 5 seconds and let asterisk plenty of time to determine is this is a fax call or if your extension is a fax only extension (i.e you don't expect to receive any voice calls on that number) explicitly disable the echo canceller using the zapec(off) command in the dialplan.

[PRI_INCOMING]

exten => _XXXX,1,NoOP(Incoming call)
exten => _XXXX,n,answer()
;give the caller something to hear while we wait.
exten => _XXXX,n,Ringing
exten => _XXXX,n,Wait(5)

This was not a fax but a normal call so let's answer it
exten => _XXXX,n,Dial(SIP/,30,r)
exten => _XXXX,n,Hangup()

;This is were we land if asterisk detects a fax call
exten => fax,1,Goto(in_fax,s,1)

;FAX
[in_fax]
;turn the echo canceler off for this channel
exten => s,1,ZapEC(off)
;call the zap channel the fax is connected to
exten => s,n,Dial(ZAP,60,r,tT)
exten => s,n,Hangup

One drawback of having a single line handling both fax and voice is that detection can not be 100% accurate.
This can be either because Asterisk did not detect correctly the CNG or the that calling fax was not sending a CNG at all.

A quick note here is that all of the above are true for zaptel based channels (pri and pstn cards) plus bristuffed (bri) zaptel.
I am not 100% sure how other channels (Sangoma for example) handle this.

And one last advice.
To make the quality of of faxes even better make sure that you have the ECM mode of the fax turned on both for transmition and reception.

Get the md5 hash of a file from Python

2010-02-04T16:18:00.003+02:00

Python has a module called haslib that provides secure hashes and message digests.
To get the md5 hash of a file all you have to do is this

import hashlib

in_file=open('path/to/file','rb').read()
hashlib.md5(in_file).hexdigest()

Life imitating art ?

2010-02-01T11:07:00.001+02:00

With the pending financial crisis in Greece, and with the recognition that our poticians have the *MAJOR* blame for what's coming, i still could not figure out why there are so many negative reports about the Greek economy that makes borrowing even more expensive and thus hindering even more the efforts for a recovery.
Then i remembered one of the dialogs from the movie "Sneakers"

Cosmo: Posit: People think a bank might be financially shaky.
Martin Bishop: Consequence: People start to withdraw their money.
Cosmo: Result: Pretty soon it is financially shaky.
Martin Bishop: Conclusion: You can make banks fail.
Cosmo: Bzzt. I've already done that. Maybe you've heard about a few? Think bigger.
Martin Bishop: Stock market?
Cosmo: Yes.
Martin Bishop: Currency market?
Cosmo: Yes.
Martin Bishop: Commodities market?
Cosmo: Yes.
Martin Bishop: Small countries?

And these come out of movie script back in 1992...
Life imitating art ?

Reverting an SVN commit

2010-01-25T19:05:00.004+02:00

Here is a one liner that helps a lot when trying to revert to an old commit after you realized that you have foobared your repository.

svn merge -c -R repository

where -R is the revision number you want to revert to.
What happens is that if R is negative it will be regarded as an inverse merge and the commit will be removed instead of added.

so

svn -c -99 http://svn.somewhere.com/trunk

will revert you trunk to revision 99

P.S Don't forget to commit after the merge if you want the reversal to be permanent

Using SAPI 4/5 voices with text to speech in Asterisk

2010-01-23T14:30:00.003+02:00

For a while now i have been working on TTS (text to speech) on Asterisk for a startup i am running with two friends.
One of my biggest problems was that the available voices for Greek in linux are very limited,sounding like robots with a laryngitis problem in some cases.
And from what i understood (or actually hear) with the exception of English most of the other languages had similar problems.
Windows has a large number of commercial "voices" that use the SAPI interface like the AT&T natural voices, Nuance, and Loqundo, but i could not find any solution that would allow me to use them with Asterisk.
Plus i wanted something that could easily scale to hundreds of channels and be low costs or free as in beer.

So i used pyTTS and web2py to create, in essence, a web service that given a string returns a file with the spoken text.
This service runs on a Windows machine and provides a very simple API to allow selection of the language,voice and fine tuning that might be required.
Once the service is invoked it returns a wav file with the spoken text.

A simple python script on the asterisk side provides a library, called ast-SAPI, that handles the communication with the web service and returns a wav file.
it pretty much works the same way as flite when its told to create wav files.

At the moment the ast-sapi gets called by a Python AGI script.
The Python AGI receives the text and settings from the Asterisk dialplan, calls the library and then streams the resulting wav.
Also got an asterisk app build using flite's asterisk app as a template,to do the same from the dialplan and avoid the AGI. It works but there are some issues that need to be addressed.

The benefits of this solution are many.

I get good quality voices to use with asterisk and since this is similar to a web service is easy to scale and spread the load to many machines.

Its still in a prototype stage, needs more work and i already spotted some places where things need to be changed, but overall seems to work.

web2py - Getting the admin panel to work remotely

2010-01-18T16:56:00.004+02:00

I have spend sometime looking at the web2py framework the past few days, as i was looking for a python based framework for a project.
My first impression is that it has some nice features but as with all "frameworks" the learning curve can be steep.

After going through the manual over the weekend i decided to take the plunge and install it in a dev machine.
This is a vm machine running Ubuntu server, so there is no local browsing capabilities everything is done over the network.
Got web2py installed, run the server and then tried to access the admin interface from my remote workstation and got this.

Admin is disabled because unsecure channel

According to the documentation the admin and appadmin only work from localhost and remotely via ssl (ssh tunnel or https)
Looking at what it takes to config Apache to test-run web2py i decided i need another simpler solution.

I found this blog post that had instructions on how to get web2py going with a self-signed certificate and gave it go.
(Note that there are some typos in instructions given in the blog, but its easy to figure them out)

So i created the certificates and tried to access web2py from https this time.
No i was getting a strange error in Firefox

SSL received a record that exceeded the maximum permissible length.

(Error code: ssl_error_rx_record_too_long)

The problem was that web2py was still sending http and not https to the browser, so probably the https was not working

I used the command line switches of web2py and provided the exact path to the certificates

python web2py.py -a pass -i 192.168.1.3 -p 8000 -c /etc/ssl/self_signed/server.crt -k /etc/ssl/self_signed/server.key

and got a new message form web2py

WARNING:root:OpenSSL libraries unavailable. SSL is OFF

It turned out that Openssl was installed on the dev machine but the python bindings were not.
so issuing a

sudo aptitude install python-openssl

solved this and remote access to web2py's admin console was available over https.

Solving Asterisk DTMF callerid issues - or why i love Open Source

2009-12-30T00:36:00.014+02:00

I have been busy the last few days with an issue that has come up, with a new batch of GSM FCT's we need to interface with some of our deployed Hermes e-IPBX units.

Hermes e-IPBX is a solution we have worked on for sometime now and basically is a build-from-scratch linux distro tailored for use with Asterisk as a PBX.
It's a "compact" solution, requiring around 32MB of flash for the entire system including a custom web based UI we have developed.

For a number of reasons that would take a lot of time to explain (and make excellent material for another blog post) our current version of Hermes e-ipbx uses Asterisk 1.2 and not the latest stable 1.4 or 1.6

The problem we faced, was that we could not get callerid working with these new FCT units.
As it turned out these units were using dtmf and not fsk to pass the callerid info between the first and second ring.

So at the begging we though, "hey that's not a problem, asterisk supports dtmf for callerid, all we have to do is change the cidsignalling variable in zapata.conf to use dtmf."
We could not be more wrong...

No matter what configuration we tried in zapata.conf, asterisk was throughing errors and we could not get the callerid info.

-- Starting simple switch on 'Zap/1-1'
Dec 28 00:07:29 ERROR[3896]: callerid.c:276 callerid_feed: fsk_serie
made mylen < 0 (-1)
Dec 28 00:07:29 WARNING[3896]: chan_zap.c:6627 ss_thread: CallerID feed
failed: Success
Dec 28 00:07:29 WARNING[3896]: chan_zap.c:6671 ss_thread: CallerID
returned with error on channel 'Zap/1-1'
-- Executing Wait("Zap/1-1", "5") in new stack
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 9 on Zap/1-1
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 1 on Zap/1-1
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 0 on Zap/1-1
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 5 on Zap/1-1
Dec 28 00:07:31 DEBUG[3896]: chan_zap.c:4907 __zt_exception: Exception
on 14, channel 1
Dec 28 00:07:31 DEBUG[3896]: chan_zap.c:4092 zt_handle_event: Got event
Ring Begin(18) on channel 1 (index 0)
Dec 28 00:07:32 DEBUG[3896]: chan_zap.c:4907 __zt_exception: Exception
on 14, channel 1
Dec 28 00:07:32 DEBUG[3896]: chan_zap.c:4092 zt_handle_event: Got event
Ring/Answered(2) on channel 1 (index 0)
Dec 28 00:07:32 DEBUG[3896]: chan_zap.c:4441 zt_handle_event: Setting
IDLE polarity due to ring. Old polarity was 0
Dec 28 00:07:34 DEBUG[3896]: pbx.c:1548
pbx_substitute_variables_helper_full: Function result is '"" <>'
-- Executing NoOp("Zap/1-1", "CALLERID="" <>") in new stack
Dec 28 00:07:36 DEBUG[3896]: chan_zap.c:4907 __zt_exception: Exception
on 14, channel 1
Dec 28 00:07:36 DEBUG[3896]: chan_zap.c:4092 zt_handle_event: Got event
Ring Begin(18) on channel 1 (index 0)

As you can see asterisk does capture part of the callerid (which is 2114019105 in this case) *after* the initial first ring, but misses the rest of it (last 4 digits).

Trying to figure out what the issue was, the first though was that the dtmf send by the device was somehow distorted,
What was need it, was a way to capture what the device was sending and check it.
So i decided to used one of what i call, asterisk's hidden super-weapons, ztmonitor.
ztmonitor is a utility that comes with asterisk and allows you to monitor a Zap (aka Dahdi) channel for signal level and also save a 'raw' image of what comes in or goes out through that interface.
It has proven a valuable tool in the past when we were trying to solve some echo issues a customer had.

The following command can be used to record any given zap channel, mixing input and output streams in a single file

ztmonitor "channel number" -f "file_name.raw"

ztmonitor has also options to capture input and output streams in different files and also get the stream without echo canceling applied.

Once you have captured the data, you can convert them to a wav file for further processing using sox.

sox -r 8000 -s -w -c 1 "file_name.raw" "file_name.wav">

What that gives you is a wav file of both streams ready for further processing and investigation.

First thing to do is to have a visual view of the captured signal.

There are several open source tools but i use Audacity for this task, as it provides several interesting features.

Following is what the captured streams looked like when loaded in Audacity.

By measuring the different parts of the stream the following info was gathered.

After the 1st ring ends, there is a 640ms delay before the dtmf starts.
Dtmf pulses have a length of 84ms and there is inter-digit delay of 120ms.

Things looked "normal",there are a few glitches in the captured stream but nothing too big and the level of noise was not big, so next we had to test if the dtmf pulses were valid.

And the tool of choice for this kind of work is none other than multimon.

Multimon can decode a variety of digital transmission modes commonly found on UHF radio using the soundcard or a wav file as input and DTMF is one of the supported.

It's also pretty straight forward to use

multimon -a DTMF -t wav ./streamtx.raw.wav

would produce an output like that.

multimod (C) 1996/1997 by Tom Sailer HB9JNX/AE4WA
available demodulators: POCSAG512 POCSAG1200 POCSAG2400 EAS AFSK1200 AFSK2400 AFSK2400_2 HAPN4800 FSK9600 DTMF ZVEI SCOPE
Enabled demodulators: DTMF
DTMF: 2
DTMF: 1
DTMF: 1
DTMF: 4
DTMF: 0
DTMF: 1
DTMF: 9
DTMF: 1
DTMF: 0
DTMF: 5

That proved that the dtmf send by the device was OK, so where is the problem ?

After looking at source code and searching for people with similar problems, it was clear that dtmf callerid in asterisk 1.2 is "partially" working.
What partially means, is that there are basically 3 ways callerid is send.
One is between the gap of the first and second ring, second is by signaling a polarity reversal before the first ring and as i found out in some countries callerid is send before the ring without any signalling (i.e polarity reversal)
Also in the case dtmf is used to send the callerid a start-of-callerid is send before the number in the means of a letter (which is different depending on country)

In Asterisk 1.2 when the callerid is send with dtmf there is only support for the polarity reversal method and not the other two.
The good thing was, that a lot of Indian telcos where using dtmf between the first and second ring so there was a patch available for 1.2 that never made it in the svn, but got incorporated in 1.4 and 1.6.

So i got the patch,removed some of the debug lines and rebuilded chan_zap.

Now i could get the last 8 digits of the callerid the FCT send, but some reason i was missing the initial two...
It looked like the dtmf detection routine was kicking-in a bit too late, missing the first 2 dtmf digits.
After a bit more of head-scratching i remembered there was definition in the zaptel card driver (wctdm.c) that was defining the time the line would take to settle after the ring.

#define DEFAULT_RING_DEBOUNCE 64 /* Ringer Debounce (64 ms) */

Setting this to 32 ms solved the issue of the two first missing digits.

The whole procedure took several days, till we managed to solve the issue, but consider the alternative if this was a closed source system, where we would have to file a bug report to the company supplying the software.
We would probably had to wait for an answer for much longer provided they had enough interest (and resources) to fix the issue.

Setting up Android SDK on Ubuntu for the Samsung Galaxy

2009-11-01T10:48:00.008+02:00

On my quest to get the Samsung Galaxy a new "ROM" to fix the issues mentioned in my last post, i installed the GalaxyHero ROM which is a custom (or "cooked") ROM.
The GalaxyHero does offer solutions to many of the issues i mentioned with the original Galaxy firmware and even the IIE update.
As usual when dealing with new devices, firmware updates of this nature,are a good way to "brick" your device so i took my time Googling, reading and taking notes of the actions required for the update, which among other things give you root access to the phone.

The first problem i encountered after downloading and installing the Android SDK and the Eclipse plugin on my 64bit Ubuntu desktop is that the phone was not recognized by the SDK.
Typing in the SDK tools dir

adb devices

came up with an empty list.

After a bit more search here is what you need to do to get your Samsung Galaxy to connect with adb on Ubuntu and allow debugging on the phone.

On the phone, in Settings/Applications/Development, check the box "USB debugging".

Then add a new udev rule for the phone
Create a file using your favorite text editor in /etc/udev/rules.d/11-android.rules
and add the following line

SUBSYSTEM=="usb_device", SYSFS{idVendor}=="04e8", MODE="0666"

Please note that the USB vendor ID is not the same for Samsung and other Android phones (aka HTC) and most of the documentation i found refers to HTC's id.

Then type this:

sudo chmod a+rx /etc/udev/rules.d/11-android.rules
sudo /etc/init.d/udev restart
./adb kill-server

The above will make the phone "recognizable" by Ubuntu,but still if you try connect with adb it does not work.
The problem is that the adb shipped with the Android SDK up to this writting (Nov 1st 2009) does not work with the Galaxy.

To overcome this you need a patched version of adb
More details and the source to build the adb yourself can be found in the German Galaxy forum

cd to your Android SDK tools dir

rename the original adb

mv adb orig-adb

download,unzip and make executable the patched adb

wget http://floe.butterbrot.org/external/adb.gz
gunzip adb.gz
chmod +x adb

then start the adb server

./adb start-server

see if the phone is recognized

./adb devices
List of devices attached
I7500c0xVS8PQ4H device

Next i will post a few simple steps to get the GalaxyHero ROM installed even without the adb

My first week with an Android phone - Samsung Galaxy i7500

2009-10-25T11:40:00.008+02:00

It's almost a week since I got my first Android based phone a Samsung Galaxy i7500 and so I decided to write a few things of what has happened these last few days.
But before the “dirty” tech details a few words about the process that got me there.

For a few months now I have been looking for a new phone,partly because my (t)rusty Qtek 9000, Windows Mobile 6.1 phone has started showing its age,partly because I was interested in looking into mobile phone development in the new platforms that have been around for a while.
As you expect the candidates were 2.
iPhone 3G(S) from Apple and an Android based phone with the iPhone having some advantage.

So I started looking around the net to see what is need it to get into programming these 2 platforms and what the dev communities say.
(I had some experience in mobile device programming both in the Microsoft Mobile and Linux, on the Neo Freerunner phone sometime in the past, but not with the new technologies that have come up since.)

To my “surprise” I realized that in order to get into iPhone development you had to be on a Mac platform (at least when I was doing my Googling, things change fast) and you had to have Apple's “blessing” if you wanted to get your application in the Apple store... Hmmm

No one can deny that Apple has some really good products and technologies but after I have “seen the light” with Open Source I really dislike the vendor lock-in.
Don't get me wrong Apple fans, as I said, the products are great but if another company (say Microsoft) had pulled the same “tricks” Apple is pulling on Palm for example, people with pitchforks would be camped outside the Microsoft HQ asking for heads to roll...

So after deciding not to go the Apple way, the next question was... “which Android based phone ?”

A lot more Googling and visits to the local wireless operator shops (I am in Greece) to find the phone.
The result.
Most of the Android phones in the (Greek) market were HTC based and hovering in the 500 Euro range, but what caught my eye was the Samsung i7500.
The AMOLED screen looked fantastic and the specs were great, (although the RAM was a bit on the “low side”) and had all the bits and pieces to use it both as a dev platform and “normal” phone.
The only thing I missed (at that time) was the keyboard, which was a great additions and the main reason I have kept my Qtek for so long.

So I got the phone, and started my journey into the world of Android and Samsung i7500.

The good stuff first...

The AMOLED screen is really great :)
The Android Market (Thank you Apple for the idea ) is really cool. I wondered how we lived without “application stores” so long :)
The design of the phone was great although "plastic" in finish.

And the bad stuff...

Power management with the default firmware sucks, big time...
There is no VPN support (at least with the Cupcake version the phone was shipped)
Tethering is not supported
The 5 MPixel camera has some serious lag issues
The default browser has file uploads disabled for “security reasons”
If you try to sync your Outlook contacts you need...Google (mail,calendar etc)
The control software (Samsung New PC Studio) shipped with the phone runs only on Windows and DOES NOT support firmware updates of the phone (what the Samsung people were thinking ???!!!)

Let's see them in more detail

After charging overnight and started using the phone, with all systems up (wifi,3G,bluetooth,GPS) I noticed that the battery was draining fast. At the beginning I thought it was because it was a new battery and LiIon batteries need a few charging cycles before the “settle” to their peak.

After a couple more days and several charging cycles I decided to shutdown all the wireless and leave only the GSM part of the phone on. Same thing.
Did more Googling, found out that others had the same problem and the problem was the firmware and it had to be updated..
But wait... The Samsung New PC Studio says I have the latest firmware (and does not even recognize the phone when going for an update)
More Googling...
The version of the PC Studio Samsung ships with the phone.... can not updated it.... need newer from the German Samsung side...
Download, install, reboot,phone recognized this time, but no new firmware....Dito again.
More Googling....
Need a 3rd party program and a firmware image people have literally hacked out of the Samsung site.
Done it, and phone works as it should (3 days now with no re-charge)
/me Thanks Samsung for having to spend a day looking for all these.

Next task....

Move my contacts from the Windows Mobile based phone to the Android one...
(Still trying to)
Well it looks that unless you use Google services and upload all your contacts and calendar data there, you are in for the long ride...

Tried to take some photos with the 5 Mpixel camera the phone has...
Disappointed... The camera lag is such that you have to press the shutter button a couple of seconds in advance... Forget those fast moving fotos...

Ok managed to take some pictures, let's try to upload them to Twitter using the web application of TwitXL...
The Upload button of the page was disabled by the buildin browser, Hmmm so I need an application like the iPhone does ? At least they have an 'explanation' that their system does not have a “filesystem” , what about Android ? (Must be the first Linux based filesystem-less system)

Wanted to do a VPN with the office, (free wifi access points are nice, but do you want all our data unencrypted over wifi ? I don't.) no luck either...
Android 1.5 (Capcake) does not have VPN support, 1.6 (Donut) has.
There is no “official” Samsung image with Donut for the moment, but the good people of the dev community have came up with solutions of their own (this is were Open Source makes all the difference)

What's the verdict after all these ?
The phone was not ready for the market... I think Samsung has pushed it out the door, to get into the “Google Wave” with little or no QC
If a non technical inclined person gets his/her hands on this phone they will return it for service in the next few days... not good, unless of course the market you are after are the “techies” like me, that can swift through pages and pages of forums and blogs looking for the little piece of info to fix it...

Android has potential, but still needs a lot of work and refinement if its going to be in “peoples” phones and not just for the techies.

BTW if you are new to the Galaxy i7500 and strangling with the same issues, this link can help (at least it did in my case)

Installing compact fluorecent lamps - A year later

2009-09-12T21:28:00.002+03:00

Last November i decided to remove all incandescent lamps from the house and replace them with low power compact fluorescent one's.
I replaced 35 100 Watt lamps with the Osram EL Dulux 20W.
http://www.osram.com/osram_com/News/General_Interest_Press/2007/Press616042.jsp

I chose those because mainly i liked the type of light they produced and decided to pay the extra cost to save some CO2 from the planet and go "green" on lighting.

A compact fluorescent lamp is much more expensive to buy than a standard incandescent one, but you save on the long run from the lower electricity consumption and the *longer* working life its supposed to have.

The Osram El Dulux 20W was rated at 10.000h which would replace 10 normal lamps, according to what is printed on the box.
Assuming an average of 6h of working time for the lamp per day , it should last for approx 1666 days or 4.5 years before it fails.

Well i can report that in less than a year, 8 from the 35 lamps have failed and need it to be replaced.
That's almost 1 in 4 (or close to 25%) loss and sure is not what i had hoped for.

I am not sure why the lamps failed, as they failed in random intervals and random places in the house, with the first staring 3-4 months after it was installed.

I don't believe that anyone from Osram will read this blog post but in the case it does, i have kept most of the failed lamps,as i was looking for a place that would accept them for recycling (yes lamps get recycled and need to be recycled as they contain some "serious" metals and not so good chemicals) and if you want i can send them back for you to test with all expenses paid.

In any case, Osram, you should check your quality as you have a really dissatisfied customer.

European Commission pushes for software patents via a trusted court

2009-06-13T17:51:00.002+03:00

First of all a warning. If part of the following text looks Greek to you,it's because it is. :)
It's part of a message i received from Konstantina Zoehrer regarding the new "initiative" to get software patents in EU, through the backdoor by use of a "trusted court".
For more info check this link.
There is a petitionalso you can sign.

Well it looks like "special interests" follow the good old saying "if you fail once, try again"
So, let's try and stop them one more time.

Greek text follows

Προτρέπουμε τους νομοθέτες μας

* να περάσουν εθνικές νομικές διευκρινίσεις για το δίκαιο ευρεσιτεχνίας για να αποκλείσουν οποιοδήποτε πατέντα λογισμικού,
* να ακυρώσουν όλους τους εγκεκριμένους ισχυρισμούς για πατέντες που μπορεί να παραβιάζονται από λογισμικό το οποίο τρέχει σε προγραμματιζόμενες συσκευές,
* επίσης να αγωνιστούν για τη διάδοση των εν λόγω κανόνων σε Ευρωπαϊκό επίπεδο, συμπεριλαμβανομένης της Ευρωπαϊκής Σύμβασης Διπλωμάτων Ευρεσιτεχνίας.

ΥΠΟΓΡΑΨΤΕ ΤΗΝ ΑΙΤΗΣΗhttp://petition.stopsoftwarepatents.eu/stats/

Διαβάστε περισσότερα εδώ http://press.ffii.org/Press_releases/European_Commission_pushes_for_software_patents_via_a_trusted_court

First results on a WiFi security survey

2009-06-02T10:51:00.003+03:00

For the past few weeks i am working on a paper for the security of the wifi networks.
Part of my research was to look at the state of security of deployed wifi AP all over Athens.
Athens is a rather large city (4+ million people leave here) and i believe that a sample of that size could be indicative of the general state if wifi all over.

The first results are rather mixed.

The good news is that the number of wifi ap's deployed is really large.
By driving through some of th main streets crossing Athens, and some of the business areas i found more that 10.000 unique MAC's of AP.
Considering that i only covered a very small portion of the Athens metropolitan area and was only driving on main roads, i estimate that the number of deployed AP will be over 200.000 to 250.000 and this is a really a very conservative estimate.

The bad news now.

More than 50% of the AP's surveyed,are using no encryption or WEP and from those using WPA 70% is using the default ESSID's set by the ISP's that provided the unit,making them easy targets for rainbow table attacks.

It looks like that most people trust the settings their ISP provides when they get their adsl IAD, or don't how to change them or simply are unaware of how wifi works and the dangers.
One ISP in particular seems to be the "king of unprotected wifi AP" in Athens as 3/4 of all the AP's with their ESSID is either open or using WEP.

I am still collecting data and working on the paper, but given the rather large number of samples i allready got i don't think the results will change much,but this is something to see.

On the funny side of things, if people do decide to change their ESSID they can be very creative :)

First results and thoughs on WPA security

2009-05-11T11:36:00.003+03:00

I have spend the last few days reading about and playing around with the various tools available for cracking WPA and here is what i came to.

Contrary to the "hype" WPA is not cracked, as WEP was.
No "fatal" design flaw has been found that can be exploited to get access to your wifi network.
The *current* and *known* (there is no way to emphasize this more) ways of getting access to WPA protected network is by the "old" way of the dictionary and/or brute force attacks and "rainbow tables".
That been send, the "weak link" in your WPA security is actual your chosen password.
If your password can be easily "guessed" (8 characters passwords with numbers (i.e bith dates) or known words like "/dev/null" :) then you might get into trouble if someone targets you.
Another thing that i realized is that by hiding your ESSID you are actually becoming an easier target to an attack.
This has to do with the fact that your ESSID is actually part of the key and the empty ESSID is on the top 10 of ESSID's people have pre-calculated rainbow tables for.

So to make your WPA wifi more secure do the following

1)Select a unique ESSID, an as attacker could not use a ready-made rainbow table, and would have to recalculate the PMK's and that can take some time, even with the help of Pyrit and some serious hardware.

2) Try to select a random password at least 20 characters long.

3) Switch to AES insted of TKIP

Getting Pyrit to work on my system and other WPA-PSK related rand

2009-05-08T10:21:00.008+03:00

Recently i started reading about WPA-PSK security, as a client was using it, in their office, and after some discussion, wanted to see how (in)secure might be.

After some reading it looks like the best way to hack a WPA-PSK based system (for the moment) is to create a rainbow table of possible PMK's (Pairwise Master Key) and then let loose tools like cowpatty and aircrack-ng.
Basically you exchange time for space, as the same PMK can be used on any AP with same SSID.

The problem is, that a PMK requires something in the range of 16.000+ rounds of SHA-1 and this requires some really big computing power.
To give you an idea of the computing power required, my quad core@3Ghz can do about 1.2K PMK's/sec,which is not that bad, but will take weeks to go over a descent word list.

Doing some more search i found that there is a program called Pyrit, that uses the power of the GPU to do some serious PMK crunching. My "vanilla" (not over clocked) Nvidia 8800 GT does 4.800/sec while other, newer Nvidia cards can reach close to 50.000/sec PMK's

UPDATE 17/5/2009
The problem mentioned below is solved in revision 99 of Pyrit
Looks like it was a CUDA 2.2 bug
I am leaving the text for 'historical reasons' but you can safely ignore the fix

So i updated my CUDA drivers and SDK (Pyrit requires CUDA 2,2), got Pyrit from the SVN, build it and run my first benchmark, using only the CPU's.
Things were good, so i moved on to build the Nvidia CUDA module for Pyrit.
The build was ok

stelios@Athena:~/pyrit/pyrit-read-only/cpyrit_cuda$ ./setup.py build
running build
running build_ext
Compiling CUDA module using nvcc 2.2, V0.2.1221...
ptxas info : Compiling entry function 'cuda_pmk_kernel'
ptxas info : Used 42 registers, 32+24 bytes smem, 12 bytes cmem[1]
Building modules...
stelios@Athena:~/pyrit/pyrit-read-only/cpyrit_cuda$ sudo ./setup.py
install
running install
running build
running build_ext
Skipping rebuild of Nvidia CUDA kernel ...
Building modules...
running install_lib
running install_egg_info
Removing /usr/lib/python2.5/site-packages/CPyrit_CUDA-0.2.3.egg-info
Writing /usr/lib/python2.5/site-packages/CPyrit_CUDA-0.2.3.egg-info

but when trying to run the benchmark again i got an error

stelios@Athena:~/pyrit/pyrit-read-only/cpyrit_cuda$ pyrit benchmark
Pyrit 0.2.3 (C) 2008, 2009 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3

The ESSID-blobspace seems to be empty; you should create an ESSID...

Failed to load CUDA-core (CUDA_ERROR_INVALID_IMAGE).
Running benchmark for at least 60 seconds...

CPU-Core (x86_64): 302.43 PMKs/s, 99.41% occupancy
CPU-Core (x86_64): 292.03 PMKs/s, 90.08% occupancy
CPU-Core (x86_64): 300.92 PMKs/s, 87.42% occupancy
CPU-Core (x86_64): 303.17 PMKs/s, 99.17% occupancy

Benchmark done. 1198.55 PMKs/s total.

For some reason the CUDA part was failing to load.

Googling about the error, i found a couple others had the same issue, so it was not just me, doing something wrong.
I emailed the author, but received no reply, so after a day started looking at the code to see where the problem comes from. Open Source rulez :)

It turned out that the module failed to load the CUDA kernel.
Pyrit "converts" the CUDA cubit module to an include file _cpyrit_cudakernel.cubin.h and then uses the CUDA API to load the kernel module.
In my case ,For some reason the _cpyrit_cudakernel.cubin.h seems to have an invalid
cuda kernel image.

So i changed the part that loads the kernel in _cpyrit_cuda.c from the include file

ret = cuModuleLoadData(&self->mod, &__cudakernel_module);

ret = cuModuleLoad(&self->mod, "/your/path/to/cubitfile/_cpyrit_cudakernel.cubin");

(P.S add the correct path to your cubit file)

that loads the cubit file directly.

That got the problem fixed and benchmark worked like a charm

stelios@Athena:~/pyrit/pyrit-read-only/cpyrit_cuda$ pyrit benchmark
Pyrit 0.2.2 (C) 2008, 2009 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3

Running benchmark for at least 60 seconds...

CUDA-Device #1 'GeForce 8800 GT': 4796.11 PMKs/s, 89.75% occupancy
CPU-Core (x86_64): 283.45 PMKs/s, 84.37% occupancy
CPU-Core (x86_64): 298.66 PMKs/s, 96.09% occupancy
CPU-Core (x86_64): 289.44 PMKs/s, 99.15% occupancy

Benchmark done. 5667.66 PMKs/s total.

Haven't looked at why the _cpyrit_cudakernel.cubin.h has a corrupted
kernel, will probably do so, during the weekend and post any patches to fix it.

Cudos to Intersys for their support

2009-04-03T12:23:00.004+03:00

Yesterday i received the replacement for my 42" Toshiba LCD TV from Intersys, the local Toshiba distributor.
Although i am kind of disappointed with Toshiba's quality, as this is the second TV from Toshiba with panel problems in 2 years, i am really pleased by the support i received from their local agent.

In both cases not only they exchange the TV but i received a newer model. (This time i got a Toshiba REGZA 42ZV555DG as a replacement for REGZA 42XV505DG i had )
Intersys acknowledging my frustration even offered to replace it with a different brand they distribute and a much larger TV (a 50" Panasonic plasma) but for a number of reasons i decided not to do so.
They have send the new TV to my home and picked up the old one, without having to pay anything, or go through hoops.

In difficult economic times like this, keeping the service and support level to your customers is really difficult, but at the same time its a very good way to keep customer loyalty.
Bad times are not going to last for ever and on when things turn better, i believe all this effort will payback.

2009-03-26T22:34:00.002+02:00

It's been almost a month since my last blog post...
Day to day work and my new pet project TwitXL are taking most of my time these days.
The Athens MediaCamp09 was a nice "distraction" also.
Met some very interesting people, with cool ideas.

I have several "half baked" posts that i need to finish and clear up so i can post them. Hope that i'll get some free time during next week.

Displaying UTF8 characters from mysql using bash

2009-02-21T12:00:00.002+02:00

I was working on a bash script that was using mysql to retrieve utf8 encoded names from a table.
Everything was smooth till i used some non English chars, like Greek and Brazilian and then i started seen a bunch of ? printed instead of the characters i was expecting.
I spend a morning looking around for a solution as i though that this was a BASH issue.
No matter what i tried the result was the same.
Then i added one more switch to the mysql query i was doing, to force mysql to output the result in utf-8, in case it was not doing so (which i was *sure* it was, as the tables were in utf-8 encoding.)

--default-character-set=utf8

After adding this all my problems were solved...

Mental note : Make sure, that when i am "sure" about something, always test it, just to be sure it works as "expected".

GoogleEarth 5 on Ubuntu 8.10

2009-02-03T11:16:00.003+02:00

Got GoogleEarth today and tried to install it on my Ubuntu Desktop. Once it was installed running the binary gave me the following error

googleearth-bin: relocation error: /usr/lib32/i686/cmov/libssl.so.0.9.8: symbol BIO_test_flags, version OPENSSL_0.9.8 not defined in file libcrypto.so.0.9.8 with link time reference

To solve this just cd to the GoogleEarth dir rename the libcrypto in something else.

stelios@Athena:~$ cd google-earth/
stelios@Athena:~/google-earth$ mv libcrypto.so.0.9.8 orig-libcrypto.so.0.9.8

Ubuntu one liner - Find out which packages are installed

2009-01-31T05:48:00.004+02:00

The following one liner can be useful if you want to see which packages are installed in your Ubuntu machine. (it should work on debian also)

dpkg --get-selections | grep -v deinstall

When getting a tcp packet from USA than a local ISP is faster...

2009-01-26T23:02:00.004+02:00

Today i decided to update our desktop linux machines to the latest and greatest Ubuntu version (8.10).
I was using it for sometime on my laptop and decided it would not break anything so it was time for an upgrade.
Since Digital-OPSiS office is in Athens, Greece,we use gr.archieve.ubuntu.com as a repo to pull updates etc.
I noticed that the download speed was not that great so i switched one of the machines to use the use the US mirror instead... and it took about 40% less time to download the same packages...
Doing a simple ping to the gr and us mirrors cleared things a bit more.

stelios@DIAS-Linux:~$ ping gr.archive.ubuntu.com
PING patroklos.noc.ntua.gr (147.102.222.211) 56(84) bytes of data.
64 bytes from patroklos.noc.ntua.gr (147.102.222.211): icmp_seq=1 ttl=58 time=197 ms
64 bytes from patroklos.noc.ntua.gr (147.102.222.211): icmp_seq=2 ttl=58 time=199 ms
64 bytes from patroklos.noc.ntua.gr (147.102.222.211): icmp_seq=3 ttl=58 time=201 ms
64 bytes from patroklos.noc.ntua.gr (147.102.222.211): icmp_seq=4 ttl=58 time=197 ms
64 bytes from patroklos.noc.ntua.gr (147.102.222.211): icmp_seq=5 ttl=58 time=194 ms
64 bytes from patroklos.noc.ntua.gr (147.102.222.211): icmp_seq=6 ttl=58 time=202 ms
64 bytes from patroklos.noc.ntua.gr (147.102.222.211): icmp_seq=7 ttl=58 time=204 ms

--- patroklos.noc.ntua.gr ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 5998ms
rtt min/avg/max/mdev = 194.562/199.653/204.150/3.273 ms

stelios@DIAS-Linux:~$ ping us.archive.ubuntu.com
PING us.archive.ubuntu.com (91.189.88.31) 56(84) bytes of data.
64 bytes from leningradskaya.canonical.com (91.189.88.31): icmp_seq=1 ttl=54 time=90.2 ms
64 bytes from leningradskaya.canonical.com (91.189.88.31): icmp_seq=2 ttl=54 time=89.9 ms
64 bytes from leningradskaya.canonical.com (91.189.88.31): icmp_seq=3 ttl=54 time=90.3 ms
64 bytes from leningradskaya.canonical.com (91.189.88.31): icmp_seq=4 ttl=54 time=89.5 ms
64 bytes from leningradskaya.canonical.com (91.189.88.31): icmp_seq=5 ttl=54 time=89.1 ms
64 bytes from leningradskaya.canonical.com (91.189.88.31): icmp_seq=6 ttl=54 time=88.6 ms

--- us.archive.ubuntu.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 88.453/89.106/90.142/0.816 ms

It looks like it takes 2.5 times more time to reach the local Athens uni, where the mirror is than the US mirror.
I was aware that there are capacity issues at the AIX the Greek providers are using for interconnecting but this is really ridiculous...

Doing a reverse ssh tunnel the embedded way

2009-01-23T14:13:00.006+02:00

Having a large number of asterisk pbx installations creates some interesting problems to people who provide support for them.
One of tha major issues is how do you get access to the PBX if it is sitting behind a firewall/nat where you have little on no control and/or has a dynamic ip.
In most cases you could forward the ssh port from the vpn/router to the the asterisk machine but there are several problem that can come up with this way.
1) More and more companies have a strict "no incoming ports open" policy.
2) Having a permanent "unrestricted" access to the pbx equipment and its logs and functions,might make some companies very skeptic.
On the other having to send a tech person to a 2 hour drive to add an extension or make a minor modification to the dialplan is rather expensive and would create a real support nightmare.

After thinking about it and discussing with our clients (and their clients), we came up with the idea of a customer triggered solution that would create some form of secure "tunnel" from the Hermes e-IPBX to the suport center of the company that is providing support for the PBX.

The idea is simple.
If a tech needs to access the PBX remotely, a person in the company calls an extension, enters a password and the the pbx creates a secure tunnel to the server of the support company giving them access.
Once support has finished, the person in the company calls again the same extension shutting down the tunnel, and everything goes back to normal.
No open ports, no "unauthorized" access.

The question then came, to what type of secure tunnel we would use.
The solution had to provide a secure login and also access to web interface (port 80) of the pbx
Since Hermes e-IPBX runs on embedded devices also it had to be something small in size.
With this in mind we started investigating two options.
A PPTP vpn and a reverse ssh tunnel.

We started with the ssh option first, as almost all of the required pieces of software where in place,as Hermes e-IPBX is using dropbear as an ssh client/server.

One very little know fact about the ssh is that on top of providing secure logins to remote hosts, it can create secure tunnels between two points and forward ports between them.
Another even less known fact is that it can create reverse tunnels, where machine A which is behind a firewall/nat can be accessed by machine B without having to change anything in machine's A firewall/nat.
For this to work, machine B must have a publicly accessible ssh server.

It goes like this

Machine A initiated a reverse ssh tunnel connection to machines B ssh serv
er.

# ssh -fNR [bind_address:]port:host:hostport] [user@]hostname

Once this is done machine B initiates a LOCAL connection to the port machine A is using as forward.

# ssh localhost -p port

As example we want to access the target node (name: localhost) from our node (name: www.support-example.org). The port we want to access is port 22 and it will be accessible from our node at port 2222.

# ssh -fNR 2222:localhost:22 www.support-example.com

After that, the you are prompted password as usual. After successful login, the command quit but it will remains in background.To access the target node, use our node to access the forwarded port (2222). So let say, if you want to ssh into the machine (because we have forwarded ssh port), we can use this command

# ssh localhost -p 2222

That is all. You now have loged in to the remote machine.

We had this example tested using our Ubuntu desktops but when we tried to implemented to the Hermes e-IPBX a few problems come up.

First, dropbear requires slightly modified commands
so the

# ssh -fNR 2222:localhost:22 www.support-example.com

becomes

# ssh -f - N -R 2222:localhost:22 www.support-example.com

Second and most important, the moment we had machine's B 22 port open (the one with the publicly accessible ssh server) we received a number of brute force ssh password guessing attacks.
Normally this would not be much of a security problem (unless of course your root password is "sex" or "god" :) but it is both annoying having to switft over large logs with failed attempts and also could work as a potential DOS attack.

The good thing about this ssh brute force attacks is that in most cases are started by "script kiddies" and the only port they scan is 22.
So if you move your ssh server to another higher port the problem pretty much disappears.
Off course as an added protection to your extra-secure-ssh-password :) it would make sense to put the machine with the ssh server on the dmz zone of your network. In the case your extra-secure-ssh-password is not so secure or a security hole is found in your ssh server.

So assuming you move the ssh server to port 3333 the command on machine A should become

# ssh -p 3333 -f - N -R 2222:localhost:22 www.support-example.com

In a following post i'll show how to merge all the above with the asterisk dialplan and create a neat support feature.

The Voltcraft energy Logger 3500 has arrived

2009-01-20T11:57:00.004+02:00

The Voltcraft energy Logger 3500 has arrived today and already had the first surprise.
As a test, I connected an extension cord which had my mobile's charger, and a BT handset charger to see how its working.
I unplugged the devices one after the other to see how much the current consumed was dropping and even when i had both of them disconnected Volcraft was registering a 1W power consumption.
I though it was weird,considering that the extension cord is passive, but then i noticed that my extension cord has a switch with a light, to show when its on or off.
Switching off the cord power consumption went to 0...
Wow 1W from the lamp of the extension cord !
Although this is close to the error range of the Voltcraft,that made me wonder how many other devices are"hidden" consumers of electricity around the house or the office.

Measuring embedded asterisk power consumption

2009-01-15T14:37:00.011+02:00

For sometime now i wanted to look at the power consumption of embedded devices running asterisk compared to a "standard" pc (if such a thing exists) and possible extend it also to measure standard PBX's.

With a first look it looks easy enough, but as with all measurements where you want an accurate result the devil is in the details.

First problem that has to be answered is "How do you get an accurate measurement of the power consumption"

There are several low-cost "energy metering devices" ( 15-20 Euros) in the market, in some cases you can find then in supermarkets also, but the accuracy they provide is rather dubious.
From a first look most could not measure power if the power was less than 5W, which is close to what a lot of embedded boards are rated at.
Also is very close to what most PC power supplies would consume in idle mode.

There are some professional equipment out there but spending 400-2500K Euros was not an option.
So after some search and reading i found a device that looks accurate enough and won't make a big dent on the budget.

This was Voltcraft Plus Energy Logger 3500 from Conrad.de

According to the specs it could measure :

Operating voltage 230 V/AC
Performance measurement display 0.1 -3500 W
Performance consumption display 0.000 - 9999 kWh
Display 3-cell with 4 positions each
Tariff range 0,000 - 9,999
Accuracy 5 - 3500 W (± 1% + 1 count)
2 -5 W (± 5% + 1 count)
less than 2W (± 15% + 1 count)

So it provides a +- 5% accuracy from 2-5 W which is rather good for a device that costs 50 Euros.

The other big advantage is that it uses an SD card to store measurements and comes with a piece of software to display the data captured.

I have placed an order for it and i expect it to arrive within next week.
That would give me enough time to figure out which devices and the way to test them.

Any ideas/suggestions/criticism is welcome.

Musopen ! copyright free (public domain) music

2009-01-12T11:26:00.002+02:00

I was looking for free (as in beer) music i could use with our HERMES e-IPBX for Music On Hold.
After some Googling i found Musopen.
Musopen is an online music library of copyright free (public domain) music.
As a footnote, If you finally decide to use it please make a donation ( i know from experience that most Open Source project need help in any form)

World PSTN Tone Database

2009-01-11T12:03:00.002+02:00

If you are interested in setting your sip phones or asterisk dialtone (or other tones) to be similar to the country you are in there is an online database with tone settings

http://www.3amsystems.com/wireline/tone-search.htm

It provides the frequency and cadence info plus it displays it with the format to be used by asterisk or sipura/linksys phones.

Updating the NVIDA CUDA driver in Ubuntu

2009-01-09T11:57:00.003+02:00

I am running Ubuntu 8.04 (Hardy) 64bit on my desktop and also have the NVIDIA CUDA installed to do some "R&D" work on accelerating asterisk codecs using the NVIDIA GPU's.
Since Ubuntu does not provide a package with the CUDA drivers i had them manually installed.

All things are great till you get a new kernel update from Ubuntu...
After the kernel update the NVIDIA driver (as expected) does not load and your X server switches to a low resolution mode.
Then you need to re-install the NVIDIA CUDA drivers and it can be a bit problematic since the installer requires X server to be shut-down.

So here is what i do

First press

CTRL-ALT-F1

to switch to a terminal window, and login

then type

sudo /etc/init.d/gdm stop

That stops the X server and you can now install the drivers.

Go to the dir where the NVIDIA-Linux-x86_64-XXX.XX-pkg2.run file is
(I am currently using NVIDIA-Linux-x86_64-177.67-pkg2.run) and write

sudo ./NVIDIA-Linux-x86_64-177.67-pkg2.run

The installer will warn you about an allready installed driver but you can ignore it.
Then it will not find a pre-build module so it will build it on the fly.

When asked about updating your X server config, answer no, as this will probably mess-it up.

Do a reboot and the NVIDA logo (with the big BETA) should come up and you should now be in the same mode as before the kernel update.

BTW you do not need to update the NVIDIA SDK or the tools.

Happy New Year

2009-01-04T12:25:00.001+02:00

Happy New Year to everyone !
Let's hope that things will get better this year, although the first sings are not that good :(

Compiling Asterisk on Ubuntu server

2008-12-12T01:04:00.004+02:00

I wanted to test something on Asterisk for a project and need it a new clean install.
Instead of setting up another machine i decided to use the power of my desktop machine ( an intel quad core with 8GBram) and go for a VM.
I decided to use QEMU for the VM as i am using it for embedded development for sometime now and it has proven a very reliable tool (plus its free :)
I created an Ubuntu server on the VM, but then i realized that the asterisk packages shipping with Ubuntu server were rather old.
So i downloaded asterisk, asterisk-addons, libpri and zaptel and started the building process.
For those of you who decide to go the same way, following are the packages you need to get asterisk with realtime (aka mysql support) to build.

sudo apt-get install build-essential speex libspeex-dev gsm-utils \
libgsm1-dev libgsmme-dev libgsm1 libgsm-tools \
libncurses5-dev ncurses-term libncurses5-dbg libsqlxx-dev doxygen

You will also need to install the linux kernel headers
so do a 'uname -r" and then

sudo apt-get install linux-headers-your-kernel-version-here

Once all these get install, follow the standard procedure to build libpri,zaptel,asterisk and asterisk-addons

Altec Telecoms - Charges beyond the "grave"

2008-12-08T13:08:00.005+02:00

To many of you who do not know it, Altec Telecoms *WAS* a Greek VOIP provider that went bankrupt a couple of months ago.
I was using their service, a pre-paid one, and i was paying using PayPal for a DID line plus the costs of the calls i was making.
When it wend bankrupt, services to customers terminated and i had something in the range of 30 Euros pre-paid to them, which i pressumed that had gone to the "never-never land"
Then i started receiving these emails
To those of you that the first part looks...Greek, don't worry it is :)

English text follows

Αγαπητέ / αγαπητή SKORONEOS,

Με το μήνυμα αυτό σας ενημερώνουμε ότι η ισχύς του i-CallIn αριθμού 2116875170 που έχετε δεσμεύσει, θα λήξει την 13/12/2008, οπότε και θα αφαιρέσουμε το ποσό των 1 € από το υπόλοιπό σας για να ανανεώσουμε την ισχύ του για τον επόμενο μήνα.

Σε περίπτωση που το υπόλοιπο του λογαριασμού σας δεν επαρκεί, ο αριθμός θα απενεργοποιηθεί (δηλαδή δεν θα μπορείτε να δέχεστε κλήσεις σε αυτόν) αλλά δεν θα αποδεσμευθεί. Θα εξακολουθήσει να είναι δεσμευμένος για εσάς για τους επόμενους 6 μήνες.

Εναλλακτικά, μπορείτε να ενεργοποιήσετε / ανανεώσετε το i-Call Unlimited για ένα μήνα επιλέγοντας αυτόν τον αριθμό, οπότε θα ανανεώσουμε την ισχύ του για ένα μήνα εντελώς δωρεάν! Περισσότερα για το i-Call Unlimited διαβάστε στο site του i-Call (http://www.i-call.gr/unlimited.aspx).

Σας ευχαριστούμε!

Dear SKORONEOS,

With this message we are informing you that your i-CallIn number 2116875170 will expire on 13/12/2008. On that date, we will deduct from your account's "wallet" the amount of 1 € to renew this number for the next month.

In case your account's balance is insufficient, the number will be deactivated (you will not be able to receive calls on it), but it will not be released from you. It will continue to be reserved for you for the next 6 months.

Alternatively, you can activate / renew i-Call Unlimited and select this number, in which case we will renew the number for an additional month free of charge! Read more about i-Call Unlimited at the i-Call site (http://www.i-call.gr/unlimited.aspx).

Thank you!

A charge notification from a bankrupt company, that has shutdown operations !!!???? WOW !

Well it looks like Altec Telecoms is now in a "zombi" state, as the people who worked there have "taken over" the installations in order to "protect the company assets until the bankruptcy court comes out with a decision" i.e get some money as compensation.

The weird think is that although i can not connect to their sip.server, they still have internet connection and can send emails and also their TDM lines are working, so when you call the DID assigned to you get directed to the voicemail...

I thank God, i decided to go with the pre-pay plan and pay with Paypal and not with a contract/monthly credit card charge, as i believe i would have seen charges been applied to the credit-card and then go though the pains of contacting the bank for canceling them....

My .2 euro cent advice to Altec Telecom customers paying with credit card... check your credit card statments.

Preparing for my son's baptizing

2008-10-23T13:12:00.003+03:00

There is a lot of activity in the house as we are preparing for my son's baptizing on November 9th.
We are expecting family from all over Greece and friends from abroad so preparations are underway for sometime now, but we are now moving to "overdrive"

OSLEC code optimization

2008-10-23T12:28:00.003+03:00

While working on my first CUDA app, i had a look at OSLEC as i though it would be a good candidate for parallel execution..
OSLEC is an Open Source echo canceler, written by David Row and used by asterisk.
After profiling the code,I found 2 places that i thought could benefit from some optimization. These were 2 loops that OSLEC spends considerable time in.
The first looked like that, and its part of the FIR filter in OSLEC

int i;
int offset1;
int offset2;

fir->history[fir->curr_pos] = sample;

offset2 = fir->curr_pos;
offset1 = fir->taps - offset2;
y = 0;
for (i = fir->taps - 1; i >= offset1; i--)
y += fir->coeffs[i]*fir->history[i - offset1];
for ( ; i >= 0; i--)
y += fir->coeffs[i]*fir->history[i + offset2];

By unrolling it it becomes like that

int i;
int offset1;
int offset2;
int position;

fir->history[fir->curr_pos] = sample;

offset2 = fir->curr_pos;
offset1 = fir->taps - offset2;
y = 0;
// for (i = fir->taps - 1; i >= offset1; i--)
// y += fir->coeffs[i]*fir->history[i - offset1];

i=fir->taps - 1;
while ( i >= offset1 )
{
position=i-offset1;
y += fir->coeffs[i]*fir->history[position];
y += fir->coeffs[i-1]*fir->history[position-1];
y += fir->coeffs[i-2]*fir->history[position-2];
y += fir->coeffs[i-3]*fir->history[position-3];
y += fir->coeffs[i-4]*fir->history[position-4];
y += fir->coeffs[i-5]*fir->history[position-5];
y += fir->coeffs[i-6]*fir->history[position-6];
y += fir->coeffs[i-7]*fir->history[position-7];

y += fir->coeffs[i-8]*fir->history[position-8];
y += fir->coeffs[i-9]*fir->history[position-9];
y += fir->coeffs[i-10]*fir->history[position-10];
y += fir->coeffs[i-11]*fir->history[position-11];
y += fir->coeffs[i-12]*fir->history[position-12];
y += fir->coeffs[i-13]*fir->history[position-13];
y += fir->coeffs[i-14]*fir->history[position-14];
y += fir->coeffs[i-15]*fir->history[position-15];

i=i-16;
}

and guess what,after recompiling there was 10% speed increase in OSLEC execution.
Then i moved to the second part that OSLEC spends a lot of time, the coefficient calculation.
Unrolling this loop gave roughly another 10% speed increase.
Roughly 20% speed increase in total, that's not bad :)

To verify that the increase were not an x86 "fluke" i added the patches to the oslec code in E-IPBX and recompiled the WARP version.
(WARP is a powerpc based PBX appliance from PIKA that we are working on)

I kept the original speedtest program and build 2 more adding each patch.
Bellow are the results first the unmodified speetest,and then with the first patch and the last with both patches.

root@e-ipbx:/$ speedtest

Testing OSLEC with 128 taps (16 ms tail)
CPU executes 527.63 MIPS
-------------------------

Method 1: gettimeofday() at start and end
601 ms for 10s of speech
31.71 MIPS
16.64 instances possible at 100% CPU load
Method 2: samples clock cycles at start and end
31.71 MIPS
16.64 instances possible at 100% CPU load
Method 3: samples clock cycles for each call, IIR average
cycles_worst 140815 cycles_last 3535 cycles_av: 4036
32.29 MIPS
16.34 instances possible at 100% CPU load

root@e-ipbx:/$ ./speedtest2

Testing OSLEC with 128 taps (16 ms tail)
CPU executes 528.91 MIPS
-------------------------

Method 1: gettimeofday() at start and end
497 ms for 10s of speech
26.29 MIPS
20.12 instances possible at 100% CPU load
Method 2: samples clock cycles at start and end
26.29 MIPS
20.12 instances possible at 100% CPU load
Method 3: samples clock cycles for each call, IIR average
cycles_worst 100679 cycles_last 3046 cycles_av: 3933
31.46 MIPS
16.81 instances possible at 100% CPU load
root@e-ipbx:/$ ./speedtest3

Testing OSLEC with 128 taps (16 ms tail)
CPU executes 527.67 MIPS
-------------------------

Method 1: gettimeofday() at start and end
485 ms for 10s of speech
25.59 MIPS
20.62 instances possible at 100% CPU load
Method 2: samples clock cycles at start and end
25.59 MIPS
20.62 instances possible at 100% CPU load
Method 3: samples clock cycles for each call, IIR average
cycles_worst 142386 cycles_last 2824 cycles_av: 2781
22.25 MIPS
23.72 instances possible at 100% CPU load

Not bad for a few lines of code :)

A super-computer in your desktop

2008-10-17T21:18:00.004+03:00

Today i decided to spend sometime and get NVIDIA CUDA working on my desktop machine ( A quad Intel Q6600@2.40GHz. with 8 GB of Ram running in 64bit mode)
I had worked with CUDA about a year ago, when it was at the early beta stage but things have moved on a lot since then.
Setting my GeForce 8800 GT on Ubuntu 8.04 took a bit longer than "usual" due to the fact that 8.04 is not supported "out of the box"
Luckily there are some very detailed pages that helped and in less than an hour, CUDA example programs were happily running.
The GeForce 8800 GT is rated at 350+ GFLOPS which is pretty impressive (consider that Cray-2 could do a "meager" 1.9 GFLOPS)
The only problem (as with a lot of the hardware these days) is that there is very little software to take advantage of it.
My reason of getting CUDA to work is to explore the possibilities of accelerating various pieces of code/applications i am interested in.
Echo canceling (OSLEC for example) and transcoding for asterisk, are high on the list and based on the experience i had of porting asterisk to an FPGA there could be some very interesting results.

Let's start blogging

2008-10-16T11:17:00.004+03:00

Up till now i had resisted the idea of starting a blog.
The main reason i had as my excuse, was time or to better put it, the lack of it. Trying to run a small business and a family with a new born is enough to make you wish for more hours in a day.
But the real reason was that i though of blogging as just "hype" that will run its course and sooner or later disappear.

Then i attended the OpenCoffee XV meeting in Athens and heard Jason Calacanis presentation.
At one point he mentioned something like "the more you give to the community the more you will get back" and that really got me thinking.
My company, Digital-OPSiS, is developing embedded Open Source solutions for sometime now, so you can say that we own our existence to the mentality of giving to the community.
If the people who started the Open Source movement haven't done so, probably many of the wonderful things that we now take for granted in the IT/Internet would not exist.
I call it the "avalanche effect" and seen it many times with open source projects.
Someone writes a few lines of code, posts the code online,and boom, developers from all over the world start contributing and before you know it you have a killer app in your hands (Asterisk anyone :)

So the same could be true about ideas or thoughts
If Jason had not said that phrase, probably i wouldn't have started this blog,so just think how many people could start something new,solve a problem or just feel better by things you and i take for granted or just don't mention.
That alone i think is a pretty good reason to start blogging