Despite the increase in bandwidth and stability of the Internet across the globe, large file uploads are often interrupted. Mainly when the upload is performed over a mobile network or from a remote area. The HTTP protocol doesn’t have out-of-the-box support to resume a failed upload, and also, there aren’t widely supported protocols. In this post, I will talk about how we make file uploads resumable.

In one of our projects, we are uploading and streaming video files sized from 100 to 1000MB. Sources of upload include desktop and mobile devices, with most uploads being done from the latter.

Let’s look at the problem of uploading larger files in more detail; then, we’ll look at how we used tus protocol and Cloudflare Stream to ensure a stable upload experience.

The need for resumable uploads

A user can enter an underground garage, their phone can lose connection when switching between base stations, or they can simply run out of battery when uploading a long video summary at the end of an eventful day.

A user might pause their upload voluntarily in case they need more broadband for a live stream or a video call.

For a better user experience, these use cases call for resumable upload support which is not present in the HTTP protocol out of the box.

A client uploading a file and the target server must adhere to the same protocol. To support features such as the ability to resume an interrupted upload or track upload progress - many upload clients, including paid libraries, split the file into chunks and upload the pieces one by one.

Resumable Upload Solutions

  1. Resumable.js is one of the front-end solutions that rely on the HTML5 File specification and leaves it to the developer to implement the server-side logic for reassembling uploaded chunks.
  2. Google Cloud Storage supports resumable file uploads, but if you’re not using GCS, you’d still have to come up with the server-side implementation.
  3. There are also several proprietary turnkey solutions that handle file uploads and put them either on their servers or on the cloud storage of the user’s choosing.
  4. tus - Then, there’s tus – an open protocol proposed by the team at Transloadit. It is relatively simple and easy to implement.

Using tus to upload large size files to CloudFlare

Cloudflare Stream uses tus to upload files larger than 200MB. I will use Cloudflare Stream API to show how a tus upload can be integrated.

Let’s suppose we have a web applications that uploads large video files to Cloudflare Stream, built with Node.js (Back-end) and React (Front-end).

The outline of the implementation would look something like this:

  1. Obtain Cloudflare API Tokens to be used on the Back-end.
  2. Implement a handler on the Back-end that requests an one-time upload URL from Cloudflare Stream (because only Back-end is aware of the Cloudflare API tokens).
  3. Whenever a file upload is initiated on the Front-end, request that one-time upload URL from the Back-end, and use the tus client JavaScript implementation to handle the upload.

Note: authentication between Front-end and Back-end is omitted from the sample source code.

API: Cloudflare Stream upload URL generation

// upload.controller.js

import * as UploadService from './upload.service';

// handle tus upload URL generation request
// /generate-tus-upload-request

export const handleTusUploadRequest = async (req, res) => {
  const contentLength = req.get('Upload-Length') || 0;
  if (!contentLength) {
    throw new Error('Upload-Length header expected');
  }

  const url = await UploadService.generateUploadUrl(contentLength);

  res.set('Access-Control-Expose-Headers', 'Location');
  res.set('Access-Control-Allow-Headers', '*');
  res.set('Access-Control-Allow-Origin', '*');
  res.set('Location', url);
  res.status(201).send('');
};

// upload.service.js

import axios from 'axios';
import {Buffer} from 'buffer';

const CLOUDFLARE_TOKEN = '...';
const CLOUDFLARE_ACCOUNT = '...';
const CLOUDFLARE_URL_EXPIRY_SEC = 60;
const CLOUDFLARE_MAX_VIDEO_DURATION_SEC = 600;

const btoa = (value) => Buffer.from(value, 'utf8').toString('base64');

const buildRequestOptions = (videoSizeBytes) => {
  if (!CLOUDFLARE_ACCOUNT || !CLOUDFLARE_TOKEN) {
    throw new Error('Incomplete Cloudflare stream configuration');
  }
  const url = `https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT}/stream/?direct_user=true`;
  const expiry = new Date(new Date().getTime() + CLOUDFLARE_URL_EXPIRY_SEC * 1000).toISOString();
	return {
    url,
    method: 'POST',
    headers: {
      'Upload-Length': videoSizeBytes,
      'Upload-Metadata':
        `maxdurationseconds ${btoa(String(CLOUDFLARE_MAX_VIDEO_DURATION_SEC))},` + `expiry ${btoa(expiry)}`,
      'Tus-Resumable': '1.0.0',
      Authorization: `Bearer ${CLOUDFLARE_TOKEN}`,
    },
  };
};

const generateUploadUrl = async (videoSizeBytes) => {
  const requestOptions = buildRequestOptions(videoSizeBytes);
  try {
    const response = await axios(requestOptions);
    return response.headers.location;
  } catch (e) {
    const errors = e?.response?.data?.errors;
    // handle errors
    throw e;
  }
};

UI: File upload using the generated URL

import React, {useCallback, useState} from 'react';
import * as tus from 'tus-js-client';

const API_SERVER_URL = '...';

export default const TusUploadInput = (props) => {
  const [displayStatus, setDisplayStatus] = useState('No file selected');

  const onError = (error) => {
    const statusCode = error.originalResponse ? error.originalResponse.getStatus() : 0;
    // notify user error
  };

  const onProgress = useCallback(
    (bytesUploaded, bytesTotal) => {
      const percentage = ((bytesUploaded / bytesTotal) * 100).toFixed(2);
      if (percentage === '100.00') {
        setDisplayStatus('Upload completed');
      } else {
        setDisplayStatus(`Uploaded ${percentage}%`);
      }
    },
    [setDisplayStatus]
  );

  const onSuccess = useCallback(() => {
    setDisplayStatus('Upload completed');
  }, [setDisplayStatus]);

  const handleFileChange = useCallback(
    async (file) => {
      if (file === null) {
        // file was deleted
        setDisplayStatus('No file selected');
        return;
      }
      setDisplayStatus('Uploading');
      const upload = new tus.Upload(file, {
        endpoint: `${API_SERVER_URL}/generate-tus-upload-request`,
        retryDelays: [0, 3000, 5000, 10000, 20000],
        metadata: {
          filename: file.name,
          filetype: file.type,
        },
        onAfterResponse: (req, res) => {
          try {
            // the unique identifier of the uploaded video
            const streamMediaId = res.getHeader('Stream-Media-ID');
          } catch (e) {
            // handle errors
          }
        },
        onError,
        onProgress,
        onSuccess,
      });

      // Check if there are any previous uploads to continue.
      upload.findPreviousUploads().then(function (previousUploads) {
        // Found previous uploads, so we select the first one.
        if (previousUploads.length) {
          upload.resumeFromPreviousUpload(previousUploads[0]);
        }

        // Start the upload
        upload.start();
      });
    },
    [setDisplayStatus, onError, onProgress, onSuccess]
  );

  return (
    <>
      <FileInput onChange={handleFileChange} {...props} />
      <div>{displayStatus}</div>
    </>
  );
};

tus 2.0

The company behind the tus 1.0, seeing its wide adoption, aspires to have the protocol become a truly open standard and thus has submitted it to be published as an IETF Internet Standard.

This will probably not change the specification of the protocol significantly but will strengthen the core part of it as an IETF standard and make the extensions optional.

The official implementation of the 2.0 version will be backward compatible and support the 1.0 version.

Benefits of tus protocol

  • Open: tus is not a product but an open and straightforward protocol, with implementations available in the most popular languages, including JavaScript, .Net, Ruby, Golang, and Java on the server, and JavaScript, iOS, Android, and others on the client.
  • Proven: tus 1.0 has been around since 2013, and many large-scale companies (like CloudFlare or Vimeo) have adopted it for their APIs.
  • Great UX: tus allows users to pick up where they left off their upload, whether the upload was accidentally interrupted or voluntarily paused, even if it happened days ago.

Bonus

In the comment section of the Hacker News entry on tus published about nine years ago, there are some exciting discussions in which one of the protocol creators is taking part.

Discuss on Twitter

Related articles: