Catalin Vasile

I'm a design-focused engineer.

How to Read Multiple Files Concurrently in Go

You can speed up the process of reading files in Go by using goroutines. This short article explains how you can easily achieve this using a simple for loop.

package main

import (
	"io/ioutil"
	"log"
	"sync"
)

func getFilesContents(files ...string) map[string][]byte {
	var wg sync.WaitGroup
	var m sync.Mutex

	filesLength := len(files)
	contents := make(map[string][]byte, filesLength)
	wg.Add(filesLength)

	for _, file := range files {
		go func(file string) {
			content, err := ioutil.ReadFile(file)

			if err != nil {
				log.Fatal(err)
			}

			m.Lock()
			contents[file] = content
			m.Unlock()
			wg.Done()
		}(file)
	}

	wg.Wait()

	return contents
}

Inside the function we first initialize a WaitGroup as well as a Mutex in order to wait for our goroutines to finish as well as to keep them in check.

The length of the files is used in order to initialize a map map[string][]byte variable called contents in which we’ll store the contents of our files.

The WaitGroup’s counter is then updated with the length of our files. Once the counter will get to zero, all goroutines will be released.

The next step is to initialize a for loop and simply use ioutil.ReadFile to read the contents of our files. In this loop we also use a mutex lock to block the calling goroutine until the mutex is available again. This allows us to safely access the data across multiple goroutines.

The getFilesContents function can now be used as follows:

func main() {
	fileContents := getFilesContents(
		"filesToRead/file1.json",
		"filesToRead/file2.json",
		"filesToRead/file3.json",
    )
    
    // fileContents => map[string][]byte
}

The returned map can now be processed as you’d like.

Subscribe

Be the first to know when I publish a new article on the blog.

No spam. You can unsubscribe at any time.


Comments