https://man.liquidfiles.com
LiquidFiles Documentation

When uploading files larger than 100MB, you should divide them into 100MB pieces and upload them in chunks. There's several benefits to this including being able to resume uploads by only re-uploading missing chunks from a larger upload, to bypassing old but still present 2GB limitations in things like web proxies.

Example Request using curl

In this example, we're taking bigfile.zip, and splitting into two files: bigfile.zip.00 and bifile.zip.01 and sending them individually like this:

curl -X POST -H "Content-Type: application/json" --user "Y9fdTmZdv0THButt5ZONIY:x" --data-binary "@bigfile.zip.00" https://liquidfiles.example.com/attachments/binary_uploads?filename=bigfile.zip&chunk=0&chunks=2
curl -X POST -H "Content-Type: application/json" --user "Y9fdTmZdv0THButt5ZONIY:x" --data-binary "@bigfile.zip.01" https://liquidfiles.example.com/attachments/binary_uploads?filename=bigfile.zip&chunk=1&chunks=2

In this example there are a few things to highlight:

  • We will get the attachment JSON object only when the last piece has been uploaded.
  • The individual chunk sizes doesn't matter. If you're sending three chunks it can be two big ones and one small, three of equal size or one big, one medium and one small. It doesn't matter.
  • You can send chunks in any order that you want. as long as you number the chunks correctly. You can for instance begin by sending the second chunk with chunk=1, followed by the first one with chunk=0. It's the chunk number that will order the pieces correctly on the server.
  • You will need to make sure that you get a http response code 200 (success) after each chunk, and resend any chunks that fail accordingly. It's only when all pieces are uploaded that the server will rebuild the attachment and give you the attachment id.
  • The "filename" parameter needs to be unique for that user until the entire file has been uploaded. It's the only thing that we have to identify this file. If you try to send multiple files with the same name to the same user at the same time (the user with the api key: Y9fdTmZdv0THButt5ZONIY in this example) there will be a right mess on the server as it has no way of distinguishing between the two different files if the name is the same.

Checking what chunks have been uploaded

When chunks are uploaded, they are stored for a week before being removed (if the file was not uploaded completely). Another thing to note is that a chunk (or a file for that matter) is interrupted during transit, the half uploaded chunk (or file) will be discarded. If you want to resume uploads, you can query the server to see what chunks are available and upload any missing chunks.

Also, a final thing to note is that chunks are unique per user and per filename. We don't have an Attachment ID (that would otherwise be unique) until the file has been completely uploaded. This means that if the same user starts uploading a new file using chunks with the same filename as the old one, and if you use this API call to check what's already uploaded you may well end up completing the old file with the remains of the new file.

There's two method for checking what chunks exist, or if if a chunk a exist.

Checking Individual Chunks before each upload

This is the method that the LiquidFiles web interface is using. Before a chunk is uploaded, we query the server and if we get a "partial content" response (HTTP reponse code 206) that means that the chunk doesn't exist and we can go ahead and upload it. If the server response with an "ok" message (HTTP response code 200) that means that the chunk already exist and we can move to the next one.

Request Info
Info Value
Request URL /attachment/binary_uploads
Request VERB GET
Request Parameters
Parameter Type Description
filename String The filename of the uploaded file.
chunk Integer The individual chunk you want to test.
chunks Integer The total number of chunks.

Example Request using curl

The following example will check if the first chunk (chunk 0) already exist on the system. If the reponse is a 206 response the chunk doesn't exist, if the response is 200, the chunk already exists.

curl -w "%{http_code}\n" --user "Y9fdTmZdv0THButt5ZONIY:x" -H "Content-Type: application/json" https://liquidfiles.company.com/attachments/available_chunks?filename=bigfile.zip&chunks=10&chunk=0

206

Or a more complete (bash) example how this would actually be used:

if [[ `curl -w "%{http_code}\n" --user "ojubiigauS2TxTy4ovk6Ph:x" -H "Content-Type: application/json" https://liquidfiles.company.com/attachments/available_chunks?filename=bigfile.zip&chunks=10&chunk=0` -eq 206 ]]; then

  curl -X POST --user "ojubiigauS2TxTy4ovk6Ph:x" -H "Content-Type: application/json" --data-binary "@bigfile.zip.00" https://liquidfiles.company.com/attachments/binary_upload?filename=bigfile.zip&chunks=10&chunk=0
fi

So we first do a GET request for the filename, chunks and chunk to see if the chunk exist and if it doesn't we upload it.

Checking All Available Chunks

You can also query the LiquidFiles system for a filename (for a specific user) and it will respond with a list of all available chunks for that filename.

Request Info
Info Value
Request URL /attachments/available_chunks
Request VERB GET
Request Parameters
Parameter Type Description
filename String The filename of the uploaded file.

The response will be an array with the following response parameters:

Response Parameters
Parameter Type Description
id String The Attachment ID
size String The chunk size in bytes

Example Request using curl

curl -X GET --user "Y9fdTmZdv0THButt5ZONIY:x" -H "Content-Type: application/json" https://liquidfiles.company.com/attachments/available_chunks?filename=bigfile.zip

{"chunks":[
  {
    "id":0,
    "size":104857600
  }, {
    "id":1,
    "size":104857600
  }]
}

In this case the first 2 chunks have been uploaded, both with 100Mb size. You can now continue with chunk 3 (chunk id 2, when starting from 0) starting at 200Mb.