send referrer header with chrome.downloads api
google-chrome-extension, javascript
Solution
The `chrome.download.download()` method's first argument (`options`) can include the headers property, an array of objects:
Extra HTTP headers to send with the request if the URL uses the HTTP[s] protocol. Each header is represented as a dictionary containing the keys name and either value or binaryValue, restricted to those allowed by XMLHttpRequest.
UPDATE:
Unfortunately, as you pointed out the "restricted to those allowed by XMLHttpRequest" part makes all the different, as the `Referer` is not an allowed header for XHR either.
I messed around quite a bit with the issue, but didn't get to a satisfying solution or work-around. I did come close though, so I will present the results here in case someone finds them useful. (Besides, some of the upcoming enhancements of the HTML5 specs might actually make this a viable work-around.)
First of all, the easy and quick alternative (with one major? drawback though) is programmatically creating and "clicking" a link (`<a>` element) that points to the image's source URL.
Pros:
- Really easy and quick to implement.
- Avoids all header- and CORS-related issues (more on that later), as it happens right from within the web-page's context.
- Does not require the `chrome.downloads` API at all.
- Does not require a background-page.
Cons:
- There is no control over where the file is downloaded (you can specify the file-name though) and whether a file-dialog is displayed or not.
If you don't care about the images' being saved at the default "downloads" folder, then this approach could be the one :)
Implementation:
var suspended = false;
window.addEventListener('click', function(evt) {
if (suspended) {
return;
}
if (evt.ctrlKey && (evt.target.nodeName === 'IMG')) {
var a = document.createElement('a');
a.href = evt.target.src;
a.target = '_blank';
a.download = a.href.substring(a.href.lastIndexOf('/') + 1);
a.click();
evt.preventDefault();
evt.stopImmediatePropagation();
suspended = true;
window.setTimeout(function() {
suspended = false;
}, 100);
}
}, true);
Trying to circumvent the above solution's limitation, I tried the following process:
- When the image is Ctrl+clicked the source URL is sent to the background-page.
- The background-page opens the URL in a new tab (so the tab's origin is the same as the images - this will be important later on).
- The background-page injects some code into the newly created tab that: a. Creates a `<canvas>` element and draws the image onto it. b. Converts the drawn image to a dataURL.(*) c. Sends the dataURL back to the background-page for further processing.
- The background-page receives the dataURL, closes the previously opened tab and initiates a download, using `chrome.downloads.download()` and the received dataURL as the value of the `url` property.
(*): To prevent "Information leakage" the canvas element does not allow the conversion of an image to dataURL, unless it has the same origin as the current web-pages location. That is why it was necessary to open the image's source URL in a new tab.
Pros:
- It allows control over whether the file-dialog will be displayed.
- It provides all the conveniences offered by the `chrome.downloads` API (if they are needed or not is different matter).
- It almost works as expected :/
Cons - Caveats:
- It is quite slow, since the image has to be loaded in the new tab.
- The maximum size of the images that can be saved depend on the maximum allowed length of a URL. Although I couldn't find any "official" resource about this, based on what most sources seem to indicate, images of a few hundred MB might be within limits (this is a personal estimation and could be pretty inaccurate). Nevertheless, this is a limitation.
- The major limitation comes form the fact that canvas' `toDataURL()` returns the data at a resolution of 96dpi. So, especially for high-resolution images, this is a show-stopper. The good news: It's sibling method `toDataURLHD()` returns it (the data) at the native canvas bitmap resolution. The not-so-good news: `toDataURLHD()` is not currently supported by Google Chrome.
You can find more info on the specifications of `canvas` and its `toDataURL()` / `toDataURLHD()` methods here. Hopefully, it will be supported soon, which will bring this solution back into the game :)
Implementation:
A sample extension would consist of 3 files:
- `manifest.json`: The manifest
- `content.js`: The content-script
- `background.js`: The background-page
manifest.json:
{
"manifest_version": 2,
"name": "Test Extension",
"version": "0.0",
"offline_enabled": false,
"background": {
"persistent": false,
"scripts": ["background.js"]
},
"content_scripts": [{
"matches": ["*://*/*"],
"js": ["content.js"],
"run_at": "document_idle",
"all_frames": true
}],
"permissions": [
"downloads",
"*://*/*"
],
}
content.js:
var suspended = false;
window.addEventListener('click', function(evt) {
if (suspended) {
return;
}
if (evt.ctrlKey && (evt.target.nodeName === 'IMG')) {
/* Initialize the "download" process
* for the specified image's source-URL */
chrome.runtime.sendMessage({
action: 'downloadImgStart',
url: evt.target.src
});
evt.preventDefault();
evt.stopImmediatePropagation();
suspended = true;
window.setTimeout(function() {
suspended = false;
}, 100);
}
}, true);
background.js:
/* This function, which will be injected into the tab with the image,
* takes care of putting the image into a canvas and converting it to a dataURL
* (which is sent back to the background-page for further processing) */
var imgToDataURL = function() {
/* Determine the image's name, type, desired quality etc */
var src = window.location.href;
var name = src.substring(src.lastIndexOf('/') + 1);
var type = /\.jpe?g([?#]|$)/i.test(name) ? 'image/jpeg' : 'image/png';
var quality = 1.0;
/* Load the image into a canvas and convert it to a dataURL */
var img = document.body.querySelector('img');
var canvas = document.createElement('canvas');
canvas.width = img.naturalWidth;
canvas.height = img.naturalHeight;
var ctx = canvas.getContext('2d');
ctx.drawImage(img, 0, 0);
var dataURL = canvas.toDataURL(type, quality);
/* If the specified type was not supported (falling back to the default
* 'image/png'), update the `name` accordingly */
if ((type !== 'image/png') && (dataURL.indexOf('data:image/png') === 0)) {
name += '.png';
}
/* Pass the dataURL and `name` back to the background-page */
chrome.runtime.sendMessage({
action: 'downloadImgEnd',
url: dataURL,
name: name
});
}
/* An 'Immediatelly-invoked function expression' (IIFE)
* to be injected into the web-page containing the image */
var codeStr = '(' + imgToDataURL + ')();';
/* Listen for messages from the content scripts */
chrome.runtime.onMessage.addListener(function(msg, sender) {
/* Require that the message contains a 'URL' */
if (!msg.url) {
console.log('Invalid message format: ', msg);
return;
}
switch (msg.action) {
case 'downloadImgStart':
/* Request received from the original page:
* Open the image's source-URL in an unfocused, new tab
* (to avoid "tainted canvas" errors due to CORS)
* and inject 'imgToDataURL' */
chrome.tabs.create({
url: msg.url,
active: false
}, function(tab) {
chrome.tabs.executeScript(tab.id, {
code: codeStr,
runAt: 'document_idle',
allFrames: false
});
});
break;
case 'downloadImgEnd':
/* The "dirty" work is done: We acquired the dataURL !
* Close the "background" tab and initiate the download */
chrome.tabs.remove(sender.tab.id);
chrome.downloads.download({
url: msg.url,
filename: msg.name || '',
saveAs: true
});
break;
default:
/* Repot invalie message 'action' */
console.log('Invalid action: ', msg.action, ' (', msg, ')');
break;
}
});
Sorry for the long answer (which does not exactly propose a working solution either). I hope someone finds something useful in there (or at least saves some time trying out the same things).
Problem
I am trying to create an extension which emulates the Ctrl+Click to save image feature from Opera 12. I am using chrome.downloads.download() from my extension's background script to initiate downloads, while a content script listens for the user action and sends a message with the URL of the image to download. Everything works fine, except that on some sites such as pixiv.net, the download is interrupted and fails. Using the webRequest API, I verified that the cookie from the active tab is being sent with the download request, but no referer header is sent. I expect that this is because the site is blocking download requests from external sites. I have not been able to verify this though, as the webRequest.onError event isn't firing on a failed download. I cannot set the referer header myself, as it cannot be set through chrome.downloads, and webRequest.onBeforeSendHeaders cannot be used in its blocking form with a download request, so I can't add the headers later. Is there a way to start a download in the context of a tab so that it behaves like right-click > save as...? For clarification as to how I'm starting the downloads, here is my TypeScript code stripped down to the applicable parts. Injected script: ``` window.addEventListener('click', (e) => { if (suspended) { return; } if (e.ctrlKey && (<Element>e.target).nodeName === 'IMG') { chrome.runtime.sendMessage({ url: (<HTMLImageElement>e.target).src, saveAs: true, }); // Prevent the click from triggering links, etc. e.preventDefault(); e.stopImmediatePropagation(); // This can trigger for multiple elements, so disable it // for a moment so we don't download multiple times at once suspended = true; window.setTimeout(() => suspended = false, 100); } }, true); ``` Background script: ``` interface DownloadMessage { url: string; saveAs: boolean; } chrome.runtime.onMessage.addListener((message: DownloadMessage, sender, sendResponse) => { chrome.downloads.download({ url: message.url, saveAs: message.saveAs, }); }); ``` Update Going off of ExpertSystem's solution below, I've found a method that mostly works. When the background script receives a request to download an image, I now compare the URL's host name against a user-configured list of sites that need the workaround. If the workaround is needed, I send a message back to the tab. The tab downloads the image using an XMLHttpRequest with `responseType = 'blob'`. I then use `URL.createObjectURL()` on the blob and pass that URL back to the background script to download. This avoids any size limitations of data URIs. Also, if the XHR request fails, I force an attempt using the standard method so that the user will at least see a failed download appear. The code now looks something like this: Injected Script: ``` // ...Original code here... chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { switch (message.action) { case 'xhr-download': var xhr = new XMLHttpRequest(); xhr.responseType = 'blob'; xhr.addEventListener('load', (e) => { chrome.runtime.sendMessage({ url: URL.createObjectURL(xhr.response), filename: message.url.substr(message.url.lastIndexOf('/') + 1), saveAs: message.saveAs, }); }); xhr.addEventListener('error', (e) => { // The XHR method failed. Force an attempt using the // downloads API. chrome.runtime.sendMessage({ url: message.url, saveAs: message.saveAs, forceDownload: true, }); }); xhr.open('get', message.url, true); xhr.send(); break; } }); ``` Background Script: ``` interface DownloadMessage { url: string; saveAs: boolean; filename?: string; forceDownload?: boolean; } chrome.runtime.onMessage.addListener((message: DownloadMessage, sender, sendResponse) => { var downloadOptions = { url: message.url, saveAs: message.saveAs, }; if (message.filename) { options.filename = message.filename; } var a = document.createElement('a'); a.href = message.url; if (message.forceDownload || a.protocol === 'blob:' || !needsReferralWorkaround(a.hostname)) { // No workaround needed or the XHR workaround is giving // us its results. chrome.downloads.download(downloadOptions); if (a.protocol === 'blob:') { // We don't need the blob URL any more. Release it. URL.revokeObjectUrl(message.url); } } else { // The XHR workaround is needed. Inform the tab. chrome.tabs.sendMessage(sender.tab.id, { action: 'xhr-download', url: message.url, saveAs: message.saveAs }); } }); ``` This will, of course, fail if the image is not on the same domain as the page and the server doesn't send CORS headers, but I figure there won't be too many sites which restrict image downloads by referrer and serve images from a different domain.