ttyS0 is for a analog modem on serial port 0, so unless you are faxing across your analog line it won't be of any use to you..
Just having a read of that wiki page you've linked to (and your quote from it) and see that 1.4 only supports T38 pass through. Basically means you need a fax machine with T38 support plugged into your asterisk box to send through it.
When I setup iaxmodem using the wiki, it setup giving the same error you show in the OP. It was due to the iaxmodem config using the wrong codec. I changed it to slin and it's been working fine since..