Posted under General
Not sure if this is a good place to ask this, but I figured since my main reason to use it would be in order to upload to Danbooru, it couldn't hurt to ask.
I recently installed Gallery-DL successfully. However, I have no idea how to use it due to a lack of a solid tutorial, and the fact that I have little skill in any coding language. Could I please have some help? How do I download an artist's Twitter page, considering how unstable that site is getting...?
i've been looking to use it as well, though not for uploading on danbooru
I heard a lot about it from Danboorians (especially those who dump artists or try and save albums before they get nuked) and I can see why it's a relatively loved tool. Like any other command line tool, once you get the hang of it, it's incredibly nifty and easy to use. I've recently started using it to download all my favourite artists' works instead of doing it one by one, painfully, with Pixiv Toolkit (and then I would totally ignore older works that I'd definitely also want to save).
I recommend downloading it with Python. You get an added bonus of being able to maintain updates through the command line instead of needing to keep up to date with and vendor the executable yourself. Generally any command line tool should be installed with another unless it's like a "root" tool, like a package ecosystem and full programming language support.
When you have it, it's as simple as running this in the command line (as an example with one of my favourite artists):
gallery-dl "https://www.pixiv.net/en/users/26502313/illustrations"
This is the most basic downloading command. There are many different options and configurations you can pass to it, and sometimes need to as well for things like R18 works or if you want to download from support sites like Patreon (cookies are required otherwise you can't download locked posts). Here are some examples I've recently run for authentication:
Downloading Nvl's works to a specific folder
Note: you may be asked to set cookies if you're trying to download any R18 works, otherwise it treats you as a logged-out user
> cd D:\gallery-dl; gallery-dl https://www.pixiv.net/en/users/28172249/illustrations -D NvlDownloading qiandaiyiyu's Patreon works with cookies from Edge
Note: I'm already pathed into my desired folder here, and I have an active membership, so I need the cookies to download locked posts
> gallery-dl --cookies-from-browser edge "https://www.patreon.com/c/qiandaiyiyu/posts" -D qiandaiyiyu
No coding experience needed; these are called command line tools. You run them by calling the name of the tool's executable (gallery-dl in this case) and pass it things called "arguments". These are used to declare how you want the command running.
The first step is to create a configuration file,
The exact location depends on the host system, but for Windows the default is %APPDATA%\gallery-dl\config.json,
See here for other locations.
For a new user I recommend starting with a minimal config and expanding as necessary, refer to the following template:
{
"extractor": {
"twitter": {
"username": "YOUR_TWITTER_NAME",
"password": "YOUR_TWITTER_PASSWORD",
"cookies": {
}
},
"pixiv": {
"refresh-token": "TODO",
"cookies": {
"PHPSESSID": "TODO"
}
}
}
}Fill in the twitter username and password fields with your account. Note that it is recommended to use a separate account specifically for gallery-dl, in case it gets banned.
Sometimes username/password authentication for Twitter doesn't work, in which case you must fill out the "cookies" section instead — see here for an explanation of how to use cookies.
For Pixiv, you must follow the OAuth process described here to obtain a value for "refresh-token".
The Pixiv extractor has a quirk which according to the author may lead to some art being missed unless the "PHPSESSID" cookie value is also provided. See above for instructions regarding cookies.
This page describes the config options available for each extractor: https://gdl-org.github.io/docs/configuration.html
Once your config file has been created, you need to launch Gallery-DL. The way to do this depends on how you have installed it.
It is a command-line application so you must start by opening Command Prompt if you're using Windows. If you're using another OS I probably don't have to explain how to open a terminal.
If you have gallery-dl installed into your %PATH% (or $PATH), you should be able to run it with the following command:
gallery-dl URL
Where URL is an address to extract from such as "https://twitter.com/bipface".
If you have installed gallery-dl via a python package manager (such as pip), try this command:
python -m gallery_dl URL
If you have simply placed a gallery-dl executable into a directory without updating %PATH% (or $PATH), you'll need to specify the full path to it, such as:
"c:\program files\gallery-dl\gallery-dl.exe" URL
Once extraction has begun, it should start to download files into a subfolder of the current directory (depending on configuration).
If you run into problems, the first thing to try is running using the --verbose argument, such as:
gallery-dl --verbose URL
Which should provide more information about what the application is doing to help you diagnose the issue.
The full list of arguments may be found here, or listed by using the --help argument.
Just gonna start all the way from step 1. I'm gonna assume your are on Windows.
Setting up
Download the gallery-dl.exe executable from here and place it in an empty folder.
Authentication
Twitter in particular is barely usable if you aren't logged in, but it's also very finicky during the login process if you use external tools like gallery-dl.
While gallery-dl can handle the login process for you, I find the easiest way is to just pass it a cookie file so the program can "act" as you, which we need from your browser. For this we need an extension.
Chrome Web Store: Get cookies.txt LOCALLY
Firefox Addons: Get cookies.txt LOCALLY
Install this extension and then browse to Twitter and make sure you are logged in.
Click on the Extension and click on Export ⇓, it will download a file called x.com_cookies.txt. Place this file in the same folder as gallery-dl.exe
Note: This TXT file contains your SESSION and gives full access to your Twitter account. It's like your username and password in a single file, treat it as such.
Usage
Open the folder where you've placed gallery-dl.exe and x.com_cookies.txt. We now need to open a command line window here. Click on the File button top left and click on Open Windows Powershell
You'll see an empty dark-blue window with the location set of the folder. Your setup should look like this:
To run the program we need to type the program name first, all commands start with this:
.\gallery-dl.exe
Then we add this to tell the program to use our cookie file for authentication:
--cookies .\x.com_cookies.txt
And then we add how much we want to download as a range. Let's say we want to download the first to the fifth post:
--range 1-5
And then we tell from what account we want to download. You can just copy and paste the URL from the browser.
https://x.com/PhdPigeon
Our complete command should look like:
.\gallery-dl.exe --cookies .\x.com_cookies.txt --range 1-5 https://x.com/PhdPigeon
Press ENTER
It will create a new sub-folder and download the images in order. Congratulations!
A quick note about Powershell
Typing long commands can be annoying, but Powershell has great autocomplete support by pressing the TAB key on your keyboard.
This will also reduce the chance of typos. So if we need:
.\gallery-dl.exe
You can just type gal and hit TAB and it should automatically change it to the right command
The same goes for
.\x.com_cookies.txt
Just type x.com and hit TAB
Dang, I went to bed after posting this and came back to a lot of people talking. Nice.
I didn't realize one of my issues would be to have my accounts across art sites mapped to it. Thank you guys so so much for the good tutorial on using this program. It feels like a lifesaver compared to my original painstakingly slow method of backing up art I like.
If you intend to run it on your pc, you may skip the cookie extraction and use the --cookies-from-browser argument instead.
hdk5 said:
If you intend to run it on your pc, you may skip the cookie extraction and use the
--cookies-from-browserargument instead.
True, which was in my original draft, however in the most recent build (1.28.0) the --cookies-from-browser argument seems to fail with Chromium browsers, even if you use an elevated Administrator Powershell.
PS C:\Users\Gabriel\Desktop\GalleryDL> .\gallery-dl.exe --cookies-from-browser chrome --range 1 https://x.com/PhdPigeon [twitter][warning] cookies: [Errno 13] Permission denied: 'C:\\Users\\Gabriel\\AppData\\Local\\Google\\Chrome\\User Data\\Default\\Network\\Cookies' [twitter][error] AuthorizationError: Login required PS C:\Users\Gabriel\Desktop\GalleryDL> .\gallery-dl.exe --cookies-from-browser edge --range 1 https://x.com/PhdPigeon [twitter][warning] cookies: [Errno 13] Permission denied: 'C:\\Users\\Gabriel\\AppData\\Local\\Microsoft\\Edge\\User Data\\Default\\Network\\Cookies' [twitter][error] AuthorizationError: Login required PS C:\Users\Gabriel\Desktop\GalleryDL> .\gallery-dl.exe --cookies-from-browser firefox --range 1 https://x.com/PhdPigeon [cookies][info] Extracted 25 cookies from Firefox .\gallery-dl\twitter\PhdPigeon\1862742863634997287_1.jpg
Gallery-dl is an incredibly powerful tool, but it can be overwhelming if you are getting started and aren't used to the command line.
So I avoided anything with Python, setting up a gallery-dl.conf file, and used as many GUI steps as possible.
So thanks to WRS and Bipface, we now have three guides aiming at various levels which should be enough for people to get started.
GabrielWB said:
True, which was in my original draft, however in the most recent build (1.28.0) the
--cookies-from-browserargument seems to fail with Chromium browsers, even if you use an elevated Administrator Powershell.
You need to close the browser you're extracting cookies from. See mikf/gallery-dl/issues/4195.