Attachment Processing Information
This article provides some background information how Attachments are processed in LiquidFiles.
When a file is upload to LiquidFiles, some of the things that may happen will take a non-trivial amount of time and will be executed in the background. Typically, we don't want to let a browser wait for longer than 30s without a response so any task that can take longer than 30s we execute as a background job.
Attachment Processing Tasks (in order of processing)
File Assembly
When a file is uploaded to LiquidFiles, if it's above 100MB in size, it's recommended that you split the file into 100MB parts (100MB chunks). This will happen automatically when you use the web interface or any of the plugins, and something you have to do yourself if you use the API. When all the chunks have been uploaded LiquidFiles will reassemble the file. If for instance you have uploaded a 10GB file, there's going to be 100 x 100MB chunks. Reading and writing 10GB can depending on your disk speed take a bit of time.
Checksum calculation
If a file has been uploaded in chunks, LiquidFiles will calculate the SHA-256 and CRC32 checksum after the file has been assembled.
Virus Scanning
Virus Scanning is controlled in Admin → Configuration → Settings. If you choose to enable virus scanning (virus scanning is enabled on default) then virus scanning will be performed as a background tasks. Virus Scanning in general is one of if not the most resource intensive task in LiquidFiles.
ActionScript Scanning
You can add your own custom Attachment Upload ActionScript to perform your own custom attachment validation. If you enable ActionScript scanning, it will be performed as a background job.
Why does this matter?
When you use the LiquidFiles web interface, you can see that when a large file has been uploaded, there will be a spinner indicating that the file is still being processed, and at the end it will process a completed check mark. But, LiquidFiles will permit that you click send before the processing has been completed. The rationale here is that in the time when the message is being delivered to the recipient and before they opens it, whatever background processing that needs to be performed has had time to complete in that time.
When you develop your own API integration, it also important to know that the immediate response from the API call to upload the file cannot be trusted, in the sense that just because you get a success message from the API, that only means that the file was successfully uploaded. It does not yet say if a file is infected with a Virus, or might be blocked by an ActionScript as part of the post processing that happens after the file has been uploaded successfully.
The Quick solutions
You can poll the attachment you've uploaded using the respective Attachment Show API. When the processed boolean flag has been set to true, that means that all post processing has completed and you can check if the content_blocked flag is true or not. If processed == true and content_blocked == false that means that there will be nothing stopping the user from downloading the file (if they have the correct permissions of course).
Processing Details
Lets look further into the details of the post processing from the API, looking at a subsection of the Attachments Attributes:
| Parameter | Type | Description |
|---|---|---|
| assembled | Boolean | True if the attachment has been assembled or not (when uploading multiple chunks) |
| virus_scan_required | Boolean | True if the attachment is required to be Virus Scanned. |
| virus_scanned | Boolean | True if the attachment has been Virus Scanned. |
| virus_scanned_at | DateTime | When the Virus Scanner last scanned this attachment. |
| content_blocked | Boolean | True if the attachment has been blocked (by the Virus Scanner or ActionScript). |
| content_blocked_message | String | If the attachment has been blocked, this will display the reason. |
| actionscript | String | If the attachment needs to be scanned by an ActionScript Upload script, this will list the name of that script. |
| actionscript_scanned | Boolean | True if the ActionScript scan has been performed. |
| actionscript_scanned_at | DateTime | When the ActionScript was executed. |
| processed | Boolean | True when the attachment has completed all required Assembly, Checksum, Virus and ActionScript scanning. |
| processed_at | DateTime | When the Attachment completed all required processing. |
| available | Boolean | True if the file is available for download or not. Please note that this is user context sensitive. If a file is required to be virus scanned and you have a setting in Admin → Configuration → Settings that Local Users can download files that are not virus scanned, then if the current user is a local user, this available will be true regardless if the virus_scan has completed or not. For external users or if Local Users are not permitted to download files that haven't been virus scanned, this will be false unless the virus scan has completed with no virus detected. |
You can see assembled, virus_scanned and actionscript parameters as well as processed, available and content_blocked parameters more meta parameters.
The assembled flag means in the strict context that the file has been assembled and the checksum has been calculated. Before an attachment has been assembled, it's not accessible regardless of other settings. When the attachment is assembled, it will be accessible for Admins with permission to access user data to download the files.
The virus_scanned and actionscript_scanned parameters will be set if you've enabled Virus Scanning (Admin → Configuration → Settings) and actionscript will be set if you have enbled Attachment Upload ActionScript scanning.
The processed flag is set when all required post processing has completed. So depending on if you've enabled virus scanning and Actionscript scanning, and if the file was uploaded in chunks or not, when everything has completed, processed will be set to true.
Available, as outlined in the table above, is user context sensitive, so an Admin with access to user data will have an attachment available == true as soon as the file has been assembled, where's an external user will only have available == true when processed == true and content_blocked == false.