Binary I/O

Question

How do you read/write a binary file into/from a Numeric array?

Advertisement: In 2012, I published the book A Hands-On Introduction to Using Python in the Atmospheric and Oceanic Sciences. If you're new to Python, you may find it useful!

Answer

Read From a Binary File

Since Python will read the binary file as just a stream of bytes, without any conversion into numeric types nor formatting into an array, the trick here is to use the read method in the array module to read and format the byte stream into the type and shape desired.

Say we have a raw binary file (i.e. without the prefix word Fortran unformatted appends to each logical record). This file tmp.bin contains single-precision (4-byte) floating point values on a latitude-longitude array. Each row is one latitude band, and there are 73 such bands at 144 longitude points; thus, the array is dimensioned 73 x 144. Adjacent longitudes, along the same latitude band, are contiguous in the file. We want to read the data from the file into a Numeric array data.

import Numeric as N
import array

num_lon = 144
num_lat = 73
tmpfile = "tmp.bin"

fileobj = open(tmpfile, mode='rb')
binvalues = array.array('f')
binvalues.read(fileobj, num_lon * num_lat)

data = N.array(binvalues, typecode=N.Float)

data = N.reshape(data, (num_lat, num_lon))

fileobj.close()

The mode flag opens the file for read-only access and tells Python it's a binary file (and not text). The binvalues object is set as single-precision floating point (type code 'f'), and 10512 (= 144 x 73) values are read into it. We then convert the values to the Numeric array format and shape the values into the 73 x 144 grid.

Write To a Binary File

To write out a Numeric array to a binary file, first convert the flat representation of the Numeric array into an array array, then use the tofile method in the array package to write the array out to a file. Say we have a single-precision floating point Numeric array data and we want to write it out to a file tmp.bin, the steps then would be:

import array
tmpfile = "tmp.bin"
fileobj = open(tmpfile, mode='wb')
outvalues = array.array('f')
outvalues.fromlist( data.flat.tolist() )
outvalues.tofile(fileobj)
fileobj.close()

There isn't a way to convert a Numeric array directly into an array array. Instead we first convert the Numeric array into a list (that's what the data.flat.tolist() line does) and then fill the array array with the fromlist method.

Note that Python references and stores array elements slightly differently than IDL and MATLAB. Also, in newer (at least post-November 2003) versions of array, the read method is deprecated; use the fromfile method instead.

---

Notes: Thanks to Guido van Rossum's posting and Mike Steder for help on this! This discussion applies to Python 2.2.