File descriptor exhaustion when calling FilesystemWriter::write

[Lynnesbian]

When creating a FilesystemWriter, calling push_file with many (over a thousand) files, then calling write, the process will crash due to running out of file descriptors.

I've created a small reproducer below - it depends on backhand and anyhow, and you'll need to replace the path with something that points to a folder with many files on your system.

use std::fs::File;
use std::io::{BufReader, BufWriter};
use backhand::{FilesystemWriter, NodeHeader};

fn main() -> anyhow::Result<()> {
    let mut squash_fs = FilesystemWriter::default();
    let path = "/lots/of/files";
    let header = NodeHeader {
      permissions: 0o755,
      ..NodeHeader::default()
    };
    
    for entry in std::fs::read_dir(&path)? {
      let entry = entry?;
      let file = File::open(entry.path())?;
      let reader = BufReader::new(file);
      squash_fs.push_file(reader, entry.path().file_name().unwrap(), header)?
    }
    
    squash_fs.write(BufWriter::new(File::create("out.squashfs")?))?;

    Ok(())
}

Is there a way around this, beside writing and re-opening the SquashFS filesystem after every few hundred calls to push_file?

[wcampbell0x2a]

I think what I would do to solve this would be to pre-read files that aren't that large, as otherwise the File will be open until after write. Something like this:

        if len < 1024 {
            let mut buffer = Vec::new();
            file.read_to_end(&mut buffer)?;
            let reader = BufReader::new(Cursor::new(buffer));
            squash_fs.push_file(reader, entry.path().file_name().unwrap(), header)?
        } else {
            let reader = BufReader::new(file);
            squash_fs.push_file(reader, entry.path().file_name().unwrap(), header)?
        }