Long overdue update

Its been over 4 years since my last post. Partly because i had lost the credentials to the blog, partly because i was busy running  my company and my family life.
In the meanwhile a lot of things have happened and among them a new hobby. Scuba Diving

Following in the footsteps of Linus Torvalds, i started to scuba dive and have acquired an urge for diving into shipwrecks.

You can see videos of my dives in a Youtube channel i have created. Its also work in progress and there many hours of dives that i haven't found time to edit and upload.
But as old habits never die, i soon realised that there are many areas in scuba diving that embedded technology can make a difference.
My first take is to make a low cost wifi enabled Nitrox analyser, an instrument used by divers to measure the amount of oxygen in their tanks.
I though that i could use this as an opportunity to explore some of the new chips in the market like the ESP8266 that are used mainly for IoT projects and have a great user base and also happen to be dirty cheap ;)
I have spend the last few weeks reading about the principals of operation of Nitrox analysers and the ESP8266 and have started building a prototype of a DIY  Nitrox analyser.
Its going to be an Open Source Hardware and Software project and plan to use this blog as engineering diary of all the work.
So stay tune more things are coming soon.

Fixing asterisk addons in Ubuntu Part 2

In an older post i have shown a way to fix the following error

Module 'app_addon_sql_mysql.so' was not compiled with the same compile-time options as this version of Asterisk.

which shows up if you try to load the mysql app in asterisk so you can store the cdr in a mysql db or access mysql from within your diaplan.

There seems to be a 'nicer' way to do it, using the debian package system itself and not having to copy files around.

Here is what you need to do

First cd to a temporary directory as this procedure will download some files and generate several deb files
As root (or use sudo) issue the following commands

aptitude purge asterisk-mysql
apt-get build-dep asterisk-mysql
apt-get -b source asterisk-mysql

Once the last command finishes you will have ended up with a number of deb files.
Then issue

dpkg -i asterisk-mysql_1.6.2.0-1_i386.deb 

The version number and architecture might be different so check yours

And that's all. No copying files around

Debugging Asterisk AGI in Python

Here is a one-liner that can literally save you days when developing a new asterisk AGI script in Python.

sys.stderr = open('/tmp/ast_agi_err.txt', 'w')

What it does is that it redirects STDERR to a file in /tmp so you can see the python console output after an un-handled  exception for example, that resulted in your agi script terminating early.
These kind of errors are not easy to find even with agi debug on, and can have you "scratching your head"  for hours if not days when debugging a new agi script

Solving the "Re-installation failed due to different application signatures" error when debugging Android apps

If you work on an android app from multiple workstations (i.e a laptop and a desktop) you might have encountered the following error, when trying to install an app for debugging on a device that has a version of the app,that was build on the other workstation.

Re-installation failed due to different application signatures.

Normally you have to go to Seetings->Apps and remove the application manually before adb can re-install it.

To solve this you need to do two things.

Copy the file debug.keystore which resides in your home_dir/.android directory to both (or all) the machines you debug from.

and second rebuild the app, using the new key.

Now the app will be "automagically" updated on the device without having to go over a manual uninstall.

NetMos 9845 Multiserial on Ubuntu 10.04

I have installed a NetMos 9845 Multiserial (8 port) card on my pc as i need several serial ports for a project.
Problem is Ubuntu does not recognize more than 4 ports by default.
To solve this you need to add a kernel boot parameter to instruct the kernel during boot to look for more serial devices.

With Grub2 on the machine, you can no longer edit the grub kernel parameters directly as Grub2 uses a templeting system.
So use your favorite editor and open (as root)

/etc/default/grub

there you will see a line

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

change it to

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash 8250.nr_uarts=8"

save the file and then issue the command

sudo update-grub

for the changes to take effect

Reboot your machine and check dmesg for the number of serial ports you now have
In my case it looks like this

:/etc/grub.d$ dmesg | grep ttyS
[ 0.709951] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.710235] 00:07: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 0.710400] 0000:05:01.0: ttyS4 at I/O 0xc000 (irq = 19) is a 16550A
[ 0.710477] 0000:05:01.0: ttyS5 at I/O 0xc100 (irq = 19) is a 16550A
[ 0.710551] 0000:05:01.0: ttyS6 at I/O 0xc200 (irq = 19) is a 16550A
[ 0.710626] 0000:05:01.0: ttyS7 at I/O 0xc300 (irq = 19) is a 16550A
[ 0.710700] 0000:05:01.0: ttyS1 at I/O 0xc400 (irq = 19) is a 16550A
[ 0.710775] 0000:05:01.0: ttyS2 at I/O 0xc500 (irq = 19) is a 16550A

You can also check the number and setting of the serial ports using the setserial command for every port

sudo setserial /dev/ttyS7 -a
/dev/ttyS7, Line 7, UART: 16550A, Port: 0xc300, IRQ: 19
Baud_base: 115200, close_delay: 50, divisor: 0
closing_wait: 3000
Flags: spd_normal skip_test

Getting your Android phone recognized by adb for debugging

In an older post i have shown hot to get a Samsung phone "recognized" by adb for debugging.
These were the "early days" of Android development...
Since then Android got more mature (so did adb) and now there are many companies producing Android compatible phones that developers want to use for testing/debugging their apps.
One of the main issues people have in Ubuntu is how to get the correct udev rules for their phone, as there are many sites ans post with old info or targeting specific phones.
The other problem developers face is how to use the phone in debug mode without having to run adb as root.

First thing to do is have a go at the official Android dev page
It has (most of) the instruction you need to get your phone connected and recognized.

It all boils down to creating a new udev rule file and adding something like this

SUBSYSTEM=="usb", ATTR{idVendor}=="0bb4", MODE="0666", GROUP="plugdev"

Basically you have to tell udev that devices with idVendor=="0bb4" should get read/write privileges (the MODE="0666" part) for the group "plugdev"
In this example, the vendor ID 0bb4 is for HTC. The MODE assignment specifies read/write permissions, and GROUP defines which Unix group owns the device node.

It also has an extensive list of vendor id's you can use to match your device.

In case your device manufacturer is not in that list, there is an easy way to find the vendor id of your device.
Connect your device and issue

lsusb

and you will get a list of connected usb devices and their associated vendor and device id's

in my case it looks like this

Bus 008 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 015: ID 12d1:1038 Huawei Technologies Co., Ltd.
Bus 001 Device 005: ID 0d8c:000c C-Media Electronics, Inc. Audio Adapter
Bus 001 Device 004: ID 05e3:0702 Genesys Logic, Inc. USB 2.0 IDE Adapter
Bus 001 Device 003: ID 0409:0050 NEC Corp.
Bus 001 Device 002: ID 03f0:4105 Hewlett-Packard ScanJet 4370
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Locate your device (in this case it's the Huawei Technologies Co., Ltd) and the first part of the ID 12d1:1038 (i.e the 12d1) is the idVendor you need you need to put in the udev rules

Another common problem developers face is that although the udev rules are correct the phone does not connect to adb and shows up as a number of ???????????? when issuing an "adb devices" command.

The only way to get the phone connected is to start the adb with root privileges using sudo ,which in my book is a big "NoNo"
To solve this you need to set the correct user and group in the udev rules
Here is how my udev entry looks like

SUBSYSTEM=="usb", SYSFS{idVendor}=="22b8", OWNER="your-user-name" GROUP="your-group"
SUBSYSTEM=="usb", ATTR{idVendor}=="22b8", MODE="0666"

To get your username and group issue the command

id

and look for the uid and gid entries
in my case they look like this

uid=1000(stelios) gid=1000(stelios)

Use the uid name in parenthesis in the OWNER and gid name in the GROUP, restart udev and now you will be able to access the phone from adb as a normal user without having to issue sudo

Running out of inodes

Here is the problem.
Had a gazillion of small files stored in a 1TB partition and everything was fine, then we need it to move these files to a smaller partition and guess what, we could not.
What happened is that we would run out of inodes in the new smaller partition when copying the data over.

Lesson learned ?
If you create a small partition and have a ton of small files you will run out of inodes, and if you use Ext4 and have flex_bg in features, then you can't use tune2fs to increase inodes in that partition.
So then the only option is to reformat the partition and increase the number of inodes

Some would argue that you could use an other filesystem with dynamic inode allocation, but personally I would go with ext4 over any version of riserfs,xfs whatever as its way more mature.

To avoid the problem do the following when formatting:

mkfs.ext4 -I 512 /dev/foo
tune2fs -i0 -c0 -o journal_data_writeback /dev/foo

Formatting this way will give you way more inodes than a "normal" format would as it increases the inode size from the default 256.

Canon CLC 3200 Ubuntu and Debian Drivers

If you are looking for the CUPS drivers of the Canon CLC 3200 for Ubuntu and Debian you can get them from here for both 32 and 64 bit version.

http://software.canon-europe.com/products/0010073.asp

Have tested them on Ununtu 10.04 both on 32bit and 64bit machines and works great.

Using the web2py framework on Eclipse

Here is a nice post on getting web2py to work through Eclipse for faster/better debugging

web2py is one of the best Python frameworks out there are the moment and if you are into Python or looking for a framework to develop the next "killer" web app have a look at it.

Fix asterisk addons problems for Ubuntu

UPDATE Got a new,simpler, way to do this here

There is a bug in the recent asterisk-addons package shipped with Ubuntu 10.04
The result is that all the modules that are part of the package fail to load with a message like this

WARNING[13478]: loader.c:800 load_resource: Module 'app_addon_sql_mysql' could not be loaded.

Till this get fixed by the Ubuntu people here a quick fix

sudo aptitude install asterisk-dev
wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-addons-1.6.2.1.tar.gz
tar xvzf asterisk-addons-1.6.2.1.tar.gz
cd asterisk-addons-1.6.2.1
./configure
make

sudo su

cp -a apps/*.so /usr/lib/asterisk/modules/
cp -a cdr/*.so /usr/lib/asterisk/modules/
cp -a channels/*.so /usr/lib/asterisk/modules/
cp -a formats/*.so /usr/lib/asterisk/modules/

Generate passwords with crypt and random salt

Here is a small perl script i found sometime ago to generate crypted passwords with random salt values

#!/usr/bin/perl

$plain=$ARGV[0]; ## Read the command line argument

if (!$plain){ ## No parameter pased
print "Usage: $0 plain_text_password\n";
exit;
}

## Use the Process id & time to generate the salt.
srand($$|time); # random seed
@saltchars=(a..z,A..Z,0..9,'.','/'); # valid salt chars
$salt=$saltchars[int(rand($#saltchars+1))]; # first random salt char
$salt.=$saltchars[int(rand($#saltchars+1))]; # second random salt char
$newuser = crypt ($newuser, $salt);

## Generate the encrypted password
$crypted=crypt ($plain, $salt);

print $crypted,"\n";

Τα μαύρα χάλια μας και καλό κουράγιο

Παρακολουθώ τους τελευταίους μήνες όλα αυτά που συμβαίνουν γύρω από τα οικονομικά της χώρας και πραγματικά αναρωτιέμαι ο 2 χρόνος γιός μου σε τι κατάσταση θα ζήσει.

Ξεκινήσαμε από το "η κρίση εμάς δεν πιάνει", ακούσα για λεφτά που υπήρχαν και μετά "εξαφανίστηκαν" και έγιναν κοψίματα μισθών, νέους έμμεσους φόρους (τη χειρότερη μορφή φορολόγησης) μέτρα εδώ, μέτρα εκεί, κομπίνες, ρεμούλες και κατασπατάληση χρημάτων αλλά πουθένα δεν άκουσα κάποιον από τους πολιτικούς μας να αναλαμβάνει ευθύνη για αυτά που έχουν γίνει στην χώρα τα τελευταία 40 χρόνια και μας έχουν οδηγήση σε αυτή τη θέση.

Λέω μήπως, μήπως οι κύριοι βουλευτές, υπουργοί και λοιποί "παρά το ...." (συμπληρώστε το κενό με ότι καρεκλοκένταυρο θέλετε) θα έπρεπε πρώτα να κόψουν τις δικές τους *πραγματικές* αμοιβές 50% και μετά να ζητήσουν από όλους τους υπόλοιπους να "θυσιασούν" μισθούς και λοιπά άλλα ?
Όχι ότι θα σωθούμε έτσι, (γιατί και του χρόνου, αν δεν υπάρξει διακανονισμός τους χρέους, σεισάχθια το είπαν οι αρχαίοι τα ίδια και περισσότερα θα χρωστάμε) άλλα το να βάζουν όλοι αυτοί που έριξαν τη χώρα στα "βράχια" με τις πολιτκές τους,τις αποφάσεις τους και τις λοπές ρεμούλες τα τελευταία 40 χρόνια, τους "άλλους" να την ξελασπώσουν *πάλι*, και αυτοί να κάνουν εξεταστικές και να τσεπώνουν τα 300άρια άνα συνδρίαση είναι λίγο...γυφτιά.
Από την άλλη όμως, οι γύφτοι μια χαρά άνθρωποί είναι και δεν φταίνε σε τίποτα για την σημερινή κατάσταση, οπότε μάλλον θα πρέπει να αλλάξουμε τη λέξη από γυφτιά σε "βουλεφτιά" ή "πολιτεφτιά" γιατί μόνο έτσι μπορούμε να καταλάβουμε το πραγματικό νόημα του τι συμβαίνει.

Και για να μην παρεξηγούμαστε, όχι ότι είμαι αντίθετος στη μείωση της σπατάλης στο Δημόσιο, το κόψιμο των "επιδομάτων" στους αργόσχολους του Δημοσίου (έδω καταντήσαμε να παίρνουν επίδομα για να μην παίρνουν 'φακελάκι'και καλά) από τους οποίους οι μισοί τουλάχιστον θα έπρεπε να πάρουν πόδι και να πάνε σπίτια τους.

Δυστηχώς το μόνο που θα καταφέρουν οι πολιτικοί μας θα είναι μια τρύπα στο νερό, τα νούμερα δεν κάνουν λάθος.
Αν χρωστάς παραπάνω από αυτά που παράγεις δεν υπάρχει περίπτωση ποτέ να μπορέσεις να μειώσεις το χρέος.
Στην καλύτερη περίπτωση μπορείς να το μετατοπίσεις για "λίγο αργότερα" (όπως γίνονταν συστηματικά μέχρι τώρα) αλλά το τέλος θα είναι πάντα το ίδιο. Θα πτωχεύσης.

Είμαστε σαν το ασθενή που είναι στην εντατική και οι "συγγενείς και φίλοι" τον κρατάνε στη ζωή για να μπορέσει να τους γράψει το "οικοπεδάκι" και να πάρουν ότι μπρορούν πριν το "μοιραίο", και το "μοιραίο" ίσως είναι πιο κοντά από όσο εμείς νομίζουμε.

Οπότε, καλό κουράγιο Έλληνες.

Creating OpenOffice Horizontal Lines

Following is a 'shortcut' for OpenOffice to create horizontal lines

Horizontal lines can be created in Writer by typing a specific symbol three times on a line by itself and then pressing Enter.
The choice of symbol * - = # ~ _ decides the type of line that is created.

You can find more shortcuts and other OpenOfice related stuff here

Solving fax issues in Asterisk

"Asterisk","fax" and "problem" are three words you find often in forums and threads of people looking for help.
In the age of email,and pdf there are still large number of companies relying in fax to transmit documents.
The reasons are many, but the problems implementors face are common.

Bad quality of faxes with "missing" or malformed lines, large number of failed faxes etc.
Most of this issues come from two facts.
The first is that we are trying to use a medium that was primarily designed for human voice to carry high speed analog modem signals and second,that fax standards are not so "standard".

Putting asterisk to handle faxes can be both a blessing and a curse for those two reasons.

Asterisk has the ability to detect if an incoming (or outgoing) call is a fax and do a number of things, depending if its acting as a "pass-through" or receive the fax.

The first thing Asterisk does is try to detect if the call is a fax call and shutdown the echo canceler on that channel.
For this to work, asterisk listens for the CNG tone emitted by the calling fax and if found, disables the echo canceler in zaptel.

The CNG is in the CCITT (ITU-T) Recommendation T.30 and describes how FAX calls are established.

During the call setup/establishment process, there are two tone signals that are send:

CNG and CED

The CNG signal *may* be sent from the Originating FAX machine after dialing is complete.
The CNG signal consists of the transmission of 1100 Hz for 1/2 second, followed by a 3 second silent (2100 Hz OFF) period.

The CED signal *may* be sent by the Terminating FAX machine anywhere between 1.8 to 2.5 seconds AFTER answering the call. The CED signal consists of a 2100 Hz tone that is from 2.6 to 4 seconds in duration. The CED tone is useful for disabling any echo cancellers on the line.

(notice the *may*, don't you love clearly defined standards ;)

Asterisk then tries to redirect the call flow to the 'fax' extension, if its present in the context that is currently executing in the dialplan.
There you can use an asterisk application to receive faxes or redirect to another port etc.

Recently we faced a problem with one of our clients that is using an E1 for voice and fax calls.
The calls come in through the E1 and the faxes are connected to an 8 port analog card
Some of the faxes received had quality issues and we tried to figure out what the problem was.

After a lot of head scratching and countless tries we realized two things.
The PRI card was somehow "loosing" some frames and that the echo canceler, OSLEC in this case, was not shutting down when a fax was received.
The first issue was easy to solve, as we discovered that the PRI card was sharing the same IRQ as the network card, so switching the pci slot and making sure that the PRI did not share IRQ's with any other device.

The tricky part was to figure out why the echo canceler was not shutting down when the call was a fax call.
The problem with an E1 (or T1) is that the calls don't use "fixed" channel numbers as in a pstn card for example, where you know that number X is on channel Y.
The provider is sending the call to the first available channel and its up to the dialplan to determine where the call would be routed.
Shutting down the echo canceler manually for all of the E1 channels, solved the issue of the faxes but created 'random' problems of echo to the voice channels.

So i started looking in detail (i.e the code) how asterisk detects faxes.

What i realized is that asterisk needs a 3-5 seconds *after* the call is answered to determine if this is a fax call,shutdown the echo canceler and then jump to the 'fax' priority.
If you "bridge" the call to another zap channel (or to sip channel of an ATA for example) before that, then the EC stays on.
And this was exactly what we were doing in our case.
As soon as the call come in, we bridged it to the analog card, not giving enough time to asterisk to shutdown the EC.

So the problem can be solved by adding a delay of 5 seconds and let asterisk plenty of time to determine is this is a fax call or if your extension is a fax only extension (i.e you don't expect to receive any voice calls on that number) explicitly disable the echo canceller using the zapec(off) command in the dialplan.

[PRI_INCOMING]

exten => _XXXX,1,NoOP(Incoming call)
exten => _XXXX,n,answer()
;give the caller something to hear while we wait.
exten => _XXXX,n,Ringing
exten => _XXXX,n,Wait(5)

This was not a fax but a normal call so let's answer it
exten => _XXXX,n,Dial(SIP/,30,r)
exten => _XXXX,n,Hangup()

;This is were we land if asterisk detects a fax call
exten => fax,1,Goto(in_fax,s,1)

;FAX
[in_fax]
;turn the echo canceler off for this channel
exten => s,1,ZapEC(off)
;call the zap channel the fax is connected to
exten => s,n,Dial(ZAP,60,r,tT)
exten => s,n,Hangup

One drawback of having a single line handling both fax and voice is that detection can not be 100% accurate.
This can be either because Asterisk did not detect correctly the CNG or the that calling fax was not sending a CNG at all.

A quick note here is that all of the above are true for zaptel based channels (pri and pstn cards) plus bristuffed (bri) zaptel.
I am not 100% sure how other channels (Sangoma for example) handle this.

And one last advice.
To make the quality of of faxes even better make sure that you have the ECM mode of the fax turned on both for transmition and reception.

Get the md5 hash of a file from Python

Python has a module called haslib that provides secure hashes and message digests.
To get the md5 hash of a file all you have to do is this

import hashlib

in_file=open('path/to/file','rb').read()
hashlib.md5(in_file).hexdigest()

Life imitating art ?

With the pending financial crisis in Greece, and with the recognition that our poticians have the *MAJOR* blame for what's coming, i still could not figure out why there are so many negative reports about the Greek economy that makes borrowing even more expensive and thus hindering even more the efforts for a recovery.
Then i remembered one of the dialogs from the movie "Sneakers"

Cosmo: Posit: People think a bank might be financially shaky.
Martin Bishop: Consequence: People start to withdraw their money.
Cosmo: Result: Pretty soon it is financially shaky.
Martin Bishop: Conclusion: You can make banks fail.
Cosmo: Bzzt. I've already done that. Maybe you've heard about a few? Think bigger.
Martin Bishop: Stock market?
Cosmo: Yes.
Martin Bishop: Currency market?
Cosmo: Yes.
Martin Bishop: Commodities market?
Cosmo: Yes.
Martin Bishop: Small countries?

And these come out of movie script back in 1992...
Life imitating art ?

Reverting an SVN commit

Here is a one liner that helps a lot when trying to revert to an old commit after you realized that you have foobared your repository.

svn merge -c -R repository

where -R is the revision number you want to revert to.
What happens is that if R is negative it will be regarded as an inverse merge and the commit will be removed instead of added.

so

svn -c -99 http://svn.somewhere.com/trunk

will revert you trunk to revision 99

P.S Don't forget to commit after the merge if you want the reversal to be permanent

Using SAPI 4/5 voices with text to speech in Asterisk

For a while now i have been working on TTS (text to speech) on Asterisk for a startup i am running with two friends.
One of my biggest problems was that the available voices for Greek in linux are very limited,sounding like robots with a laryngitis problem in some cases.
And from what i understood (or actually hear) with the exception of English most of the other languages had similar problems.
Windows has a large number of commercial "voices" that use the SAPI interface like the AT&T natural voices, Nuance, and Loqundo, but i could not find any solution that would allow me to use them with Asterisk.
Plus i wanted something that could easily scale to hundreds of channels and be low costs or free as in beer.

So i used pyTTS and web2py to create, in essence, a web service that given a string returns a file with the spoken text.
This service runs on a Windows machine and provides a very simple API to allow selection of the language,voice and fine tuning that might be required.
Once the service is invoked it returns a wav file with the spoken text.

A simple python script on the asterisk side provides a library, called ast-SAPI, that handles the communication with the web service and returns a wav file.
it pretty much works the same way as flite when its told to create wav files.

At the moment the ast-sapi gets called by a Python AGI script.
The Python AGI receives the text and settings from the Asterisk dialplan, calls the library and then streams the resulting wav.
Also got an asterisk app build using flite's asterisk app as a template,to do the same from the dialplan and avoid the AGI. It works but there are some issues that need to be addressed.

The benefits of this solution are many.

I get good quality voices to use with asterisk and since this is similar to a web service is easy to scale and spread the load to many machines.

Its still in a prototype stage, needs more work and i already spotted some places where things need to be changed, but overall seems to work.

web2py - Getting the admin panel to work remotely

I have spend sometime looking at the web2py framework the past few days, as i was looking for a python based framework for a project.
My first impression is that it has some nice features but as with all "frameworks" the learning curve can be steep.

After going through the manual over the weekend i decided to take the plunge and install it in a dev machine.
This is a vm machine running Ubuntu server, so there is no local browsing capabilities everything is done over the network.
Got web2py installed, run the server and then tried to access the admin interface from my remote workstation and got this.

Admin is disabled because unsecure channel

According to the documentation the admin and appadmin only work from localhost and remotely via ssl (ssh tunnel or https)
Looking at what it takes to config Apache to test-run web2py i decided i need another simpler solution.

I found this blog post that had instructions on how to get web2py going with a self-signed certificate and gave it go.
(Note that there are some typos in instructions given in the blog, but its easy to figure them out)

So i created the certificates and tried to access web2py from https this time.
No i was getting a strange error in Firefox

SSL received a record that exceeded the maximum permissible length.

(Error code: ssl_error_rx_record_too_long)

The problem was that web2py was still sending http and not https to the browser, so probably the https was not working

I used the command line switches of web2py and provided the exact path to the certificates

python web2py.py -a pass -i 192.168.1.3 -p 8000 -c /etc/ssl/self_signed/server.crt -k /etc/ssl/self_signed/server.key

and got a new message form web2py

WARNING:root:OpenSSL libraries unavailable. SSL is OFF

It turned out that Openssl was installed on the dev machine but the python bindings were not.
so issuing a

sudo aptitude install python-openssl

solved this and remote access to web2py's admin console was available over https.

Solving Asterisk DTMF callerid issues - or why i love Open Source

I have been busy the last few days with an issue that has come up, with a new batch of GSM FCT's we need to interface with some of our deployed Hermes e-IPBX units.

Hermes e-IPBX is a solution we have worked on for sometime now and basically is a build-from-scratch linux distro tailored for use with Asterisk as a PBX.
It's a "compact" solution, requiring around 32MB of flash for the entire system including a custom web based UI we have developed.

For a number of reasons that would take a lot of time to explain (and make excellent material for another blog post) our current version of Hermes e-ipbx uses Asterisk 1.2 and not the latest stable 1.4 or 1.6

The problem we faced, was that we could not get callerid working with these new FCT units.
As it turned out these units were using dtmf and not fsk to pass the callerid info between the first and second ring.

So at the begging we though, "hey that's not a problem, asterisk supports dtmf for callerid, all we have to do is change the cidsignalling variable in zapata.conf to use dtmf."
We could not be more wrong...

No matter what configuration we tried in zapata.conf, asterisk was throughing errors and we could not get the callerid info.

-- Starting simple switch on 'Zap/1-1'
Dec 28 00:07:29 ERROR[3896]: callerid.c:276 callerid_feed: fsk_serie
made mylen < 0 (-1)
Dec 28 00:07:29 WARNING[3896]: chan_zap.c:6627 ss_thread: CallerID feed
failed: Success
Dec 28 00:07:29 WARNING[3896]: chan_zap.c:6671 ss_thread: CallerID
returned with error on channel 'Zap/1-1'
-- Executing Wait("Zap/1-1", "5") in new stack
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 9 on Zap/1-1
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 1 on Zap/1-1
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 0 on Zap/1-1
Dec 28 00:07:29 DEBUG[3896]: chan_zap.c:4001 zt_handle_dtmfup: DTMF
digit: 5 on Zap/1-1
Dec 28 00:07:31 DEBUG[3896]: chan_zap.c:4907 __zt_exception: Exception
on 14, channel 1
Dec 28 00:07:31 DEBUG[3896]: chan_zap.c:4092 zt_handle_event: Got event
Ring Begin(18) on channel 1 (index 0)
Dec 28 00:07:32 DEBUG[3896]: chan_zap.c:4907 __zt_exception: Exception
on 14, channel 1
Dec 28 00:07:32 DEBUG[3896]: chan_zap.c:4092 zt_handle_event: Got event
Ring/Answered(2) on channel 1 (index 0)
Dec 28 00:07:32 DEBUG[3896]: chan_zap.c:4441 zt_handle_event: Setting
IDLE polarity due to ring. Old polarity was 0
Dec 28 00:07:34 DEBUG[3896]: pbx.c:1548
pbx_substitute_variables_helper_full: Function result is '"" <>'
-- Executing NoOp("Zap/1-1", "CALLERID="" <>") in new stack
Dec 28 00:07:36 DEBUG[3896]: chan_zap.c:4907 __zt_exception: Exception
on 14, channel 1
Dec 28 00:07:36 DEBUG[3896]: chan_zap.c:4092 zt_handle_event: Got event
Ring Begin(18) on channel 1 (index 0)

As you can see asterisk does capture part of the callerid (which is 2114019105 in this case) *after* the initial first ring, but misses the rest of it (last 4 digits).

Trying to figure out what the issue was, the first though was that the dtmf send by the device was somehow distorted,
What was need it, was a way to capture what the device was sending and check it.
So i decided to used one of what i call, asterisk's hidden super-weapons, ztmonitor.
ztmonitor is a utility that comes with asterisk and allows you to monitor a Zap (aka Dahdi) channel for signal level and also save a 'raw' image of what comes in or goes out through that interface.
It has proven a valuable tool in the past when we were trying to solve some echo issues a customer had.

The following command can be used to record any given zap channel, mixing input and output streams in a single file

ztmonitor "channel number" -f "file_name.raw"

ztmonitor has also options to capture input and output streams in different files and also get the stream without echo canceling applied.

Once you have captured the data, you can convert them to a wav file for further processing using sox.

sox -r 8000 -s -w -c 1 "file_name.raw" "file_name.wav">

What that gives you is a wav file of both streams ready for further processing and investigation.

First thing to do is to have a visual view of the captured signal.

There are several open source tools but i use Audacity for this task, as it provides several interesting features.

Following is what the captured streams looked like when loaded in Audacity.






By measuring the different parts of the stream the following info was gathered.

After the 1st ring ends, there is a 640ms delay before the dtmf starts.
Dtmf pulses have a length of 84ms and there is inter-digit delay of 120ms.

Things looked "normal",there are a few glitches in the captured stream but nothing too big and the level of noise was not big, so next we had to test if the dtmf pulses were valid.

And the tool of choice for this kind of work is none other than multimon.

Multimon can decode a variety of digital transmission modes commonly found on UHF radio using the soundcard or a wav file as input and DTMF is one of the supported.

It's also pretty straight forward to use

multimon -a DTMF -t wav ./streamtx.raw.wav

would produce an output like that.

multimod (C) 1996/1997 by Tom Sailer HB9JNX/AE4WA
available demodulators: POCSAG512 POCSAG1200 POCSAG2400 EAS AFSK1200 AFSK2400 AFSK2400_2 HAPN4800 FSK9600 DTMF ZVEI SCOPE
Enabled demodulators: DTMF
DTMF: 2
DTMF: 1
DTMF: 1
DTMF: 4
DTMF: 0
DTMF: 1
DTMF: 9
DTMF: 1
DTMF: 0
DTMF: 5

That proved that the dtmf send by the device was OK, so where is the problem ?

After looking at source code and searching for people with similar problems, it was clear that dtmf callerid in asterisk 1.2 is "partially" working.
What partially means, is that there are basically 3 ways callerid is send.
One is between the gap of the first and second ring, second is by signaling a polarity reversal before the first ring and as i found out in some countries callerid is send before the ring without any signalling (i.e polarity reversal)
Also in the case dtmf is used to send the callerid a start-of-callerid is send before the number in the means of a letter (which is different depending on country)

In Asterisk 1.2 when the callerid is send with dtmf there is only support for the polarity reversal method and not the other two.
The good thing was, that a lot of Indian telcos where using dtmf between the first and second ring so there was a patch available for 1.2 that never made it in the svn, but got incorporated in 1.4 and 1.6.

So i got the patch,removed some of the debug lines and rebuilded chan_zap.

Now i could get the last 8 digits of the callerid the FCT send, but some reason i was missing the initial two...
It looked like the dtmf detection routine was kicking-in a bit too late, missing the first 2 dtmf digits.
After a bit more of head-scratching i remembered there was definition in the zaptel card driver (wctdm.c) that was defining the time the line would take to settle after the ring.

#define DEFAULT_RING_DEBOUNCE 64 /* Ringer Debounce (64 ms) */

Setting this to 32 ms solved the issue of the two first missing digits.

The whole procedure took several days, till we managed to solve the issue, but consider the alternative if this was a closed source system, where we would have to file a bug report to the company supplying the software.
We would probably had to wait for an answer for much longer provided they had enough interest (and resources) to fix the issue.