Inputs and Outputs

Toolchest works with local or remote files. We reccomend using local or S3 files for data integrity and speed of execution, but HTTP or FTP URLs can also be used.

For all tools and file types, inputs accepts a string path or a list of paths. output_path always accepts a directory path.

Let's take a look at what it looks like to use different types of local and remote paths!

📘

You can mix and match file sources

We dive into each individual local and remote option here, but you can mix and match. Every file is handled independently, so you can use S3, FTP, and local files in the same Toolchest call.

Local files and directories

Local files are the most intuitive – you just pass normal paths directly to Toolchest. In the background, the files are transferred to and from the cloud.

inputs accepts local files or directories. If a directory is passed, all files within the directory are used as input.

output_path accepts local directories. Output files are written in this directory.

# With a local input file
toolchest.test(
    inputs="./input.txt",
    output_path="./output"
)

# With a local input directory
toolchest.test(
    inputs="./inputs/",
    output_path="./output"
)
# With a local input file
toolchest$test(
    inputs = "./input.txt",
    output_path = "./output/"
)

# With a local input directory
toolchest$test(
    inputs = "./inputs/",
    output_path = "./output/
)

Remote files

S3

S3 files are the fastest and most reliable input source. Toolchest pulls directly from the path you pass.

inputs accepts S3 URIs for a file. If you have multiple files in an S3 directory, make sure to list the directory first and pass each file as an input.

output_path accepts an S3 URI for a S3 prefix.

📘

Make sure Toolchest has access to your S3 bucket

To grant Toolchest access, see AWS Integration..

toolchest.test(
    inputs="s3://toolchest-public-examples/example.fastq",
    output_path="s3://toolchest-public-output/remote-output/"
)
toolchest$test(
    inputs = "s3://toolchest-public-examples/example.fastq",
    output_path = "s3://toolchest-public-output/remote-output/"
)

HTTP/HTTPS

🚧

HTTP and HTTPS files are dangerous!

We can't guarantee data integrity on transfer, because different servers behave differently. Make sure that the HTTP server supports GET requests with the range header. Always use a local or S3 file path if possible. Ye be warned!

inputs accepts an HTTP URL for a file. If you have multiple files in an HTTP directory, make sure to list the directory first and pass each file as an input.

output_path does not accept HTTP outputs at this time.

toolchest.test(
    inputs="https://rest.uniprot.org/uniprotkb/P48754.fasta",
    output_path="./"
)
toolchest$test(
    inputs = "https://rest.uniprot.org/uniprotkb/P48754.fasta",
    output_path = "./"
)

FTP

🚧

FTP files are dangerous!

We can't guarantee data integrity on transfer, because different servers behave differently. Always use a local or S3 file path if possible. Ye be warned!

inputs accepts an FTP URL for a file. If you have multiple files in an FTP directory, make sure to list the directory first and pass each file as an input.

output_path does not accept FTP outputs at this time.

toolchest.test(
    inputs="ftp://ftp.sra.ebi.ac.uk/vol1/fastq//SRR999/000/SRR9990000/SRR9990000.fastq.gz",
    output_path="./"
)
toolchest$test(
    inputs = "ftp://ftp.sra.ebi.ac.uk/vol1/fastq//SRR999/000/SRR9990000/SRR9990000.fastq.gz",
    output_path = "./"
)

Did this page help you?