https://man.liquidfiles.com
LiquidFiles Documentation

Sending files in chunks

This will enable you to divide and send a large file uploaded in smaller pieces. This only works with the html form based upload and works by you splitting a large file in smaller pieces (chunks) and sending the chunks individually. When completed the server will rebuild the complete file. The benefits to this is that if one upload fails, the entire file doesn't have to be retransmitted. And some devices such as Microsoft ISA and TMG proxies struggle with files larger than 2Gb. Sending files in chunks will get around this and enable files of unlimited (well, limited by disk space) file size.

Request

Request URL: /attachments
Request VERB: POST
Parameters:
  Filedata:      # The html multiplart file data
  name           # String.  The file name. This is needed because we no longer can use the filename from the
                 #          html multipart file data.
  chunk          # Integer. The current piece between 0-(number of pieces -1)
  chunks         # Integer. The total number of pieces
Response:
  Attachment ID  # String.  A unique string representing the attachment id. This will be used to reference the
                 #          files in the message.

Example Request using curl

In this example, we're taking bigfile.zip, and splitting into two files: bigfile.zip.00 and bifile.zip.01 and sending them individually like this:

#!/bin/sh

# Some nice variables
api_key="Y9fdTmZdv0THButt5ZONIY"
server="https://liquidfiles.example.com"

curl -X POST --user "$api_key:x" -F Filedata=@bigfile.zip.00 -F name=bigfile.zip -F chunk=0 -F chunks=2 $server/attachments
attachment_id=`curl -X POST --user "$api_key:x" -F Filedata=@bigfile.zip.01 -F name=bigfile.zip -F chunk=1 -F chunks=2 $server/attachments`

In this example there are a few things to highlight:

  • We will get the attachment id only when the last piece has been uploaded.
  • The individual chunk sizes doesn't matter. If you're sending three chunks it can be two big ones and one small, three of equal size or one big, one medium and one small. It doesn't matter.
  • You can send chunks in any order that you want. as long as you number the chunks correctly. You can for instance begin by sending the second chunk with chunk=1, followed by the first one with chunk=0. It's the chunk number that will order the pieces correctly on the server.
  • The script above doesn't have any error handling. You need to make sure that you get a http response code 200 (success) after each chunk, and resend any chunks that fail accordingly. It's only when all pieces are uploaded that the server will rebuild the attachment and give you the attachment id.
  • The "name" parameter needs to be unique for that user until the entire file has been uploaded. It's the only thing that we have to identify this file. If you try to send multiple files with the same name to the same user at the same time (the user with the api key: Y9fdTmZdv0THButt5ZONIY in this example) there will be a right mess on the server as it has no way of distinguishing between the two different files if the name is the same.

Checking what chunks have been uploaded

This API call requires LiquidFiles v2.5 or later.

When chunks are uploaded, they are stored for a week before being removed (if the file was not uploaded completely). Another thing to note is that a chunk (or a file for that matter) is interrupted during transit, the half uploaded chunk (or file) will be discarded. If you want to resume uploads, you can query the server to see what chunks are available and upload any missing chunks.

Also, a final thing to note is that chunks are unique per user and per filename. We don't have an Attachment ID (that would otherwise be unique) until the file has been completely uploaded. This means that if the same user starts uploading a new file using chunks with the same filename as the old one, and if you use this API call to check what's already uploaded you may well end up completing the old file with the remains of the new file.


Request

Request URL: /attachments/available_chunks
Request VERB: GET
Format: JSON
Parameters:
  name           # String.  The file name.
Response:
  chunks         # Array.   The response will be an array with chunk ID's and chunk sizes (in bytes).

Example Request using curl

#!/bin/sh

# Some nice variables
api_key="Y9fdTmZdv0THButt5ZONIY"
server="https://liquidfiles.company.com"

curl -X GET --user "$api_key:x" -H "Content-Type: application/json" $server/attachments/available_chunks?name=bigfile.zip

{"chunks":[
  {
    "id":0,
    "size":104857600
  }, {
    "id":1,
    "size":104857600
  }]
}

In this case the first 2 chunks have been uploaded, both with 100Mb size. You can now continue with chunk 3 (chunk id 2, when starting from 0) starting at 200Mb.