I've been attempting to make a XTF file reader for years. I don't know how many times I've been given a hard drive of sonar data with no map whatsoever of what I'm going to be looking at. Usually I load it into SonarWiz, export a crude mosaic, then look at it in a separate program to see where the data was taken. You could always spend several boring hours then exporting the navigations data and trying to build track lines in cad by using what I call the AutoCAD command line dump. That also takes quite awhile.
What I've always wanted is a simple script that would look at raw XTF files and generate track lines in ArcGIS. I think I have found a way.
For those that don't know XTF (eXtended Triton Format) is a popular file format for sidescan data. It is a binary file and is grouped as packets of data. The file format is open and Triton still maintains the file format.
I decided to try to use Python to read an XTF. Find the appropriate packets and then parse the navigation data. The reason I used Python is that it's somewhat readable and I could also use the code in ArcGIS to write the navigation straight to a feature class (using arcpy- and a future blog).
This is what I came up with. It's two files- a main engine that controls the loop and opening , closing and writing the files and another file that has all the functions for dealing with the packet types found in XTF files.
Dealing with binary data in Python is easier than it seems. Most Python documentation reads like it will only deal with text files but this is untrue of course. You need to do the import strict and open the file in a binary format:
import struct
mMyXtffile = open("c:\\xtffile.xtf", 'rb')
To use struct you have to read the packet in and then disassemble it using struct.unpack. Unpack requires instructions on how to take apart the read data. This is given by using letters as d,f,b,etc. These letters stand for what the chunk of binary data is d=double, f = float, b= byte. The XTF file documentation has a list of what each byte is in a packet should bt. A better description of struct is here.
For instance the first fourteen bytes in any XTF packet has the designated fields:
| Start Position Byte Number | Type | Contents |
|---|---|---|
| 0 | 2 byte hex number | magic number 0(xFACE) |
| 2 | 1 byte unsigned integer | Packet Type |
| 3 | 1 byte unsigned integer | Sub channel number (for bathymetry) |
| 4 | 2 bytes unsigned integer | Number of channels to follow (for sonar) |
| 6 | 2 bytes unsigned integer | Unused (x2)1 |
| 10 | 4 byte unsigned integer | Number of bytes in this packet |
We can use struct to determine what kind of packet we are looking at then use the appropriate unpack command to parse it. In this case we can look at the first fourteen bytes using this struct command:
def XTFFourteenBytes(fileobject):
binarydata = fileobject.read(14)
datatuple = struct.unpack('<HBBHHHL', binarydata)
return(fileobject, datatuple)
The '<HBBHHHL' part of the struct.unpack command tells the function exactly what should be in the first fourteen bytes and it returns a tuple to “datatuple” which is returned along with the open XTF to master_exploder.
Once master_exploder determines what kind of packet we are looking at it will send the file to the appropriate function for parsing. Usually that's going to be a sonar ping header. The function I have devised uses struct to look at the Ping Headers looks like :
XTFPingHeader(fileobject):
binarydata = fileobject.read(242)
datatuple = struct.unpack('<H6BHLLffL21fddHH4Bffdd4H10fLfL4BhhcI7B'
,binarydata)
return (fileobject, datatuple)
Note how much more complicated the struct.unpack command is on that one.
It is totally necessary to determine if your unpack command is written correctly. One of the tricks I have learned to use is the struct.calcsize()
>>> import struct
>>> struct.calcsize('<H6BHLLffL21fddHH4Bffdd4H10fLfL4BhhcI7B')
242
Now there are 13 different types of packets in the XTF format – including the file header and chan info packets. I took the time to build a function to read each one and double check it. It takes awhile for sure. I might post it – if I feel generous.
while xtfFileCurPosition < xtfFileSize:
xtffile, packethead = xtf_reader.XTFFourteenBytes(xtffile)
packettype = packethead[1]
bytesinpacket = packethead[6]
if packettype == 0:
#print "we have a %s packet" % packetTypeLib['0']
xtffile, xtfRetData = xtf_reader.XTFPingHeader(xtffile)
xtfdata.append(xtfRetData)
#this additional stuff will skip the chan info and data
#structures
skippackets = bytesinpacket - 256
xtffile.seek(skippackets,1)
xtfFileCurPosition = xtffile.tell()
# this portion currently writes the xcoorg and ycoord
# to a csv file
# the gis version will call xtfGIS can start writing
#geometry
ycord = xtfRetData[44]
xcord = xtfRetData[45]
#print ('%s, %s') % (xcord,ycord)
csvfile.write('{0},{1}\n'.format(xcord,ycord))
elif packettype == 1:
#print "we have a %s packet" % packetTypeLib['1']
xtffile, xtfRetData = xtf_reader.XTFNotes(xtffile)
xtfdata.append(xtfRetData)
xtfFileCurPosition = xtffile.tell()
As I just wanted a proof of concept to start I have constructed the Master_Exploder engine to skip the actual sonar data and just get the header data (including the navigation data). But you still have to skip the data what's why use a seek command on the file after calculating the remaining number of bytes in the packet.
So this isn't the best way to read a XTF file (and I've left some stuff out to just show the important stuff). I need to work on the following things:
Recent Comments