Programming fundamentals - Part 05: I/O, files and scripting

We already have made some brief contact with I/O in the programs we have made so far. For example we know that we can print text onto the terminal screen with the print() function. We also used io.read() to read data from the keyboard.

These are what is referred to as the "standard I/O". The computer terminal's text input and output are the fundamental, basic I/O mechanism that has existed since the very early days of computing and on some early computers were the only I/O devices programs would interact with.

Although standard I/O may seem pretty basic, modern operating systems are capable of I/O redirection using mechanisms like pipes where output of a program can be directed into the input of another program to execute pretty complex tasks with nothing but basic I/O commands.

To interact with standard I/O we have a few functions at our disposal. First, we have the ordinary print(), which will take a series of textual and non-textual arguments as parameters and display them and add a line break at the end. There is also a fairly similar function io.write() that does almost the same things, expect io.write() will not add a line break by default.

To demonstrate, these lines produce the same output:

print("Hello world!") 
io.write("Hello world!\n")

Hello world!
Hello world!

As you can see, we had to add "\n" to the end of the io.write() call. This is a representation of the line break character, which tells the computer to move its output to a new line and is one of many invisible control characters. If we didn't do that, whatever is outputed next would end up on the same line we were on:

io.write("Hello ")
io.write("world! ")
print("Still the same line.")

Hello world! Still the same line.

So, io.write() allows us to have a little bit more control over our output compared to print(). In many cases this won't matter, but sometimes you want to make sure text is produced exactly the way you want it to be and in those cases io.write() might make your life easier. For debugging and basic programs, print() will likely be enough though.

In order to read data from the user, we have the io.read() function. The io.read() function takes one parameter, which determines how much data it should read. The options are "*all", "*line" "*number" or a number. "*all" will direct Lua to read data until the end of a file is encountered. This might be a bit confusing, since should be reading from the terminal, not a file, but we will get back to this. For reading interactive input, the "*line" and "*number" options are a bit more useful: they will read one line of text input or a single number, respectively. If you give the function a number as a parameter, it will indicate the number of characters to be read.

So, for example, we could ask the user to provide their name like so:

#+name io.lua

io.write("Enter your name: ") 
local name = io.read("*line")

print("Hello, " .. name .. "!")

λ lua io.lua
Enter your name: Sami
Hello, Sami!

Or we we want to make a basic calculator to add two numbers:

#+name io.lua

io.write("Enter first number: ")
local a = io.read("*number")
io.write("Enter second number: ")
local b = io.read("*number")

print(a .. "+" .. b .. "=" .. a + b)

λ lua calculator.lua 
Enter first number: 1
Enter second number: 1
1+1=2

As you can see, we don't need to manually convert the input into a number, as the io.read() function does it for us when we use the "*number" argument.

Interactive I/O is useful when we are dealing with relatively small amounts of data or when our program is simply intended to be part of a larger pipeline of programs producing and filtering data. But if we want to access persistent data storage ourselves, we need to deal with files.

The good thing is that we already know how to read and write to files. Because, as it turns out, the standard I/O also consists of files, they just happen to be special files that refer to the interactive terminal by default. So, that means that io.write() and io.read() will work just as well for working with files on our disk.

However, first we need to actually tell Lua that we would like to use a specific file instead of the stdin and stdout files that are used by default. We do this with the io.input() and io.output() functions, which take the name of the file as a parameter and open it for reading and writing respectively.

Suppose we have a text file named "hello.txt" with the following content:

#+name hello.txt

Hello world!
Second line!
42

We can read the contents of the file and then write them to another file called "hello2.txt" like so:

io.input("hello.txt") 

local firstLine = io.read("*line")
local secondLine = io.read("*line")
local answer = io.read("*number")

io.output("hello2.txt")

io.write(firstLine, "\n")
io.write(secondLine, "\n")
io.write(answer, "\n")

This simple file mechanism is indeed simple and might be enough for some programs, but it does have some downsides. As you can see, the functionality operates based on setting an input and an output file that will be operated on currently. If you have to juggle with multiple files, this may become difficult to keep track of due to the problems with global state. Secondly, it doesn't allow us all of the granularity that the operating system provides for file access. For instance, the default behavior for output files is to overwrite the contents of the file, so if you would like to keep the existing data around, you would first need to read the file and write the data back before continuing to write more data.

A slightly more controlled and robust file access mechanism is also provided, at the cost of slightly increased complexity.

Instead of setting a current input and output file, it operates on the concept of file handles, which represent currently opened files. You can have essentially as many files open simultaneously as the operating system allows, which is typically a really high number. However, we should also remember to close files when we are done with them to avoid unnecessarily holding files open.

You create a file handler with the io.open() function, which takes the name of the file and a "mode", which determines how we want to interact with the file. The mode can be "r" for reading, "w" for writing, "a" for appending and "b" for binary files.

In order to read and write data related to the opened file, we use methods provided by the handle, instead of the global io.read() and io.write() functions. Similarly, the file handle has a method for closing the handle when we are done with the file.

To read the same "hello.txt" file and writing to "hello2.txt" file, we would construct our program like so:

local inputFile = io.open("hello.txt", "r") -- file opened for reading  

local firstLine = inputFile:read("*line")
local secondLine = inputFile:read("*line")
local answer = inputFile:read("*number")

inputFile:close()

local outputFile = io.open("hello2.txt", "w")  -- file opened for writing

outputFile:write(firstLine .. "\n")
outputFile:write(secondLine .. "\n")
outputFile:write(answer .. "\n")

outputFile:close()

The way we are calling the methods on the handles might look a bit strange. We are used to calling various functions with a dot instead of a colon. So, what gives?

A piece of code like inputFile:read() is actually just a bit of syntactic sugar. All of the methods that become available on a file handle when we create one are just functions that take the file handle itself as their first argument. You could imagine the function to look a bit like this:

function read(handle, arguments...)
   -- Code to read data from a file
end

The file handle itself is just a table that contains the functions for interacting with the file, so something like this:

inputFile = {
    -- Some special data related to the file
    -- ....
    -- Methods:
    read = read
    -- so on...
}

So, each time we want to call the inputFile.read() function, we could do so like this:

local data = inputFile.read(inputFile, "*line")

However, that is pretty repetive, so Lua provides a method syntax that automatically provides the reference to the table as the first argument if we call the function with the colon syntax:

local data = inputFile:read("*line")

We will be dealing with methods on tables more when we look into object-oriented programming, but for now this should be enough for you to be able to at least use libraries that provide a method-oriented interface, such as the file I/O library of Lua.

Whether you use the simple I/O mechanism using io.input() and io.output() or the more complete version based around io.open() is up to you, but I would generally suggest opting for the latter. File handles allow you to deal with files as ordinary values in your program, allowing them to be passed to functions as parameters, stored in tables and so on, which helps create programs that are easier to reason about and also avoiding the limitation of a single input and output file at a time.

Another way of doing I/O is for our programs to use services provided by the operating system. This allows things like running other programs or accessing extended OS functions for date and time information, environment variables and file management.

The operating system functionality in Lua is found in the os module. We won't go through all of its capabilities, since some of them can be a bit more complex than what we have time for. You can always read about all of it under the Lua documentation if you are interested.

What we are interested in is being able to run commands and programs, which we can do with the os.execute() function. The os.execute() function provides access to the operating system terminal, and essentially most things we can run in the operating system command shell should be possible to run via os.execute().

For example, if we want to see a listing of the files in the current directory, we can call the operating system command for directory listing:

os.execute("dir") -- we use "dir" here, since it should work on Windows, Mac and Linux

...
programming-fundamentals-01.org
programming-fundamentals-02.org
programming-fundamentals-03.org
programming-fundamentals-04.org
programming-fundamentals-05.org
programming-fundamentals-06.org
...

One annoyance is that we cannot directly get access to the output of the command via os.execute(). It would be possible to instruct to OS to send the output into a file and read it from there or use io.popen() to get a file handle to the program's output.

However, if your main goal is to simply run a command in response to some data in your program, os.execute() should allow you to do that. This would allow you to create Lua programs that can do pretty complex interactions with the OS to manage various processes and workflows. This kind of programming is also often referred to as "scripting".

Now you should have a decent idea of how to make programs that interact not only with a user using the terminal, but also with files and the operating system shell. A lot of productive applications are now possible, including automation of various tasks and maintaining stored data for longer-term processing.

At this point you basically have a grasp of what I would call the absolute basics. Most basic computation tasks can be achieved with just basic arithmetic, loops, conditionals, compound data and I/O.

However, naturally we are not stopping yet. Next time we will have a look at computing at a slightly lower level by simulating how a computer works under the hood. After that we will begin exploring ways to better manage increasing complexity of our programs and start delving into more advanced topics.

See you next time!