Donmai

New Pixiv image links?

Posted under Bugs & Features

Toks

Ars said:

Edit: Oh man, I think they might have gotten rid of the old URL style entirely. For newer images, anyways.

evazion said:

Fuck me, you're right. I can't get old URLs to work on any new images.

None of these work now:

Hm... well we could still replace new -> old style post sources, only problem is that the post's source will then point to a 404 page. But since Danbooru will direct users who click the source link to the html mode=medium page anyway instead of the 404 image page, that doesn't seem like a huge problem.

For now though we might want to wait a little while in case Pixiv makes more changes soon.

Reply
Toks

evazion said:

I guess. I'd just hate to have to use nonworking old-style URLs in the source field just to make the artist finder and source:pixiv/username searches to work. It could cause problems for RaisingK's md5 mismatch script or any other scripts that need to download the file from the original source.

Yeah, I guess RaisingK would have to make two requests to get the real image: one to Pixiv's api or something to get the new style url and a second for the image itself. Not ideal but it should work.

Reply
Toks

Ars said:

Okay, for now, there's a workaround for getting the original images off manga posts:

http://i2.pixiv.net/c/1200x1200/img-master/img/2014/09/30/23/12/11/46276161_p0_master1200.jpg

Chop off _master1200 and replace c/1200x1200/img-master with img-original

http://i2.pixiv.net/img-original/img/2014/09/30/23/12/11/46276161_p0.jpg

Replace .jpg with .png or .gif if needed.

A simple script that replaces/generates the links should suffice until pixiv fixes this. (I can't into coding, else I'd make one myself. ;_;)

Well, IF they even plan on fixing it.

It would make sense for Danbooru to change it to the original url whenever someone tries to upload the 1200px one. But since Pixiv might still change this here's a more immediate measure: userscript. This changes the 1200px manga images into originals. (Only on mode=manga_big pages, mode=manga will still display the samples like they always have.)

Reply
Toks

evazion said:

I guess. I'd just hate to have to use nonworking old-style URLs in the source field just to make the artist finder and source:pixiv/username searches to work. It could cause problems for RaisingK's md5 mismatch script or any other scripts that need to download the file from the original source.

Alternatively we could add a new pixiv_moniker field to the posts table (like how we have pixiv_id) to specifically store the artist's username for that kind of search. Then we can just use the new-style urls in the source. But then we'd have to index the new field too, and any more post indexes is very likely going to break uploads.

Reply
evazion

Toks said:

Alternatively we could add a new pixiv_moniker field to the posts table (like how we have pixiv_id) to specifically store the artist's username for that kind of search. Then we can just use the new-style urls in the source.

I was thinking about that approach recently. The idea could be generalized by having an artist_username_string on posts. The value would look like "site:username" (so for example: "pixiv:nounanka" or "deviantart:shilesque"). We could use that field to support efficient source:site/username searches for any site, not just Pixiv.

Taking it further: artist entries could have an artist_usernames_string too. For abubu it might look something like "pixiv_id:103703 pixiv:nounanka twitter:abubu_nounanka fc2:nounanka". Then artist finding could be done by taking a source URL, figuring out the username from it (which the "fetch source data" code already knows how to do), then searching artist_usernames_string for an artist entry with that username. This way artist finding could work even for URLs that don't contain an artist name, since it wouldn't rely on URL matching at all.

I think this approach would be a lot more robust than URL matching. Rewriting the artist finder to support this would admittedly be a fair amount of work, though.

But then we'd have to index the new field too, and any more post indexes is very likely going to break uploads.

Yeah, I don't know what could be done about that.

Reply
Toks

evazion said:

I was thinking about that approach recently. The idea could be generalized by having an artist_username_string on posts. The value would look like "site:username" (so for example: "pixiv:nounanka" or "deviantart:shilesque"). We could use that field to support efficient source:site/username searches for any site, not just Pixiv.

Taking it further: artist entries could have an artist_usernames_string too. For abubu it might look something like "pixiv_id:103703 pixiv:nounanka twitter:abubu_nounanka fc2:nounanka". Then artist finding could be done by taking a source URL, figuring out the username from it (which the "fetch source data" code already knows how to do), then searching artist_usernames_string for an artist entry with that username. This way artist finding could work even for URLs that don't contain an artist name, since it wouldn't rely on URL matching at all.

I think this approach would be a lot more robust than URL matching. Rewriting the artist finder to support this would admittedly be a fair amount of work, though.

That would be a nice improvement, and it would also help fix issue #2011 (although we'd probably still have to hard-code each individual site we want that issue fixed for).

evazion said:

Yeah, I don't know what could be done about that.

One possible way would be to use delayed jobs to populate the index instead of doing it as soon as a post is uploaded, like I suggested to fix a similar problem in issue #1335. Though race conditions might be more likely to happen here than with issue #1335, it would need some testing.

Reply
ElectricSheep

Toks said:

It would make sense for Danbooru to change it to the original url whenever someone tries to upload the 1200px one. But since Pixiv might still change this here's a more immediate measure: userscript. This changes the 1200px manga images into originals. (Only on mode=manga_big pages, mode=manga will still display the samples like they always have.)

This may not work if the 1200px version has an extension that differs from the full-size version.

Reply
Anonymous9000

How does one use the GreaseMonkey script in Firefox? I know I have the script downloaded, but I don't see any option to run it anywhere, let alone a Pixiv site. There's no option to execute it in the dropdown list when I make the Tools dropdown menu show up.

Reply
ElectricSheep

Toks said:

Whoops, didn't realize the 1200px versions were forced to be jpgs. Fixed version

Can it happen that the uploader uses different file formats for different pages? I think the API may not cover that case

Anonymous9000 said:

How does one use the GreaseMonkey script in Firefox? I know I have the script downloaded, but I don't see any option to run it anywhere, let alone a Pixiv site. There's no option to execute it in the dropdown list when I make the Tools dropdown menu show up.

Toks' script only applies to the &mode=manga_big image pages of individual manga items.

I updated my userscript to provide similar behavior via inline-expansion on all pages: https://greasyfork.org/scripts/3091-pixiv-infinite-scroll-download-links

Kinda annoying to deal with those changes since I have to deal with both old and new formats. And things tend to break left and right because I rewrite the DOM and they also updated their stylesheets :/

Reply
ZipFile

ElectricSheep said:

Can it happen that the uploader uses different file formats for different pages? I think the API may not cover that case

http://spapi.pixiv.net/iphone/manga.php?illust_id={ID}

z905844 said:

Probably not related but it seems pixiv is doing something that prevents thumbnails from being similar searched- or even giving out image contextual menus when right clicked.

._layout-thumbnail:after, ._work:before {display: none !important;}

Updated

Reply
Ars

Looks like they fixed manga uploads; originals are now displayed correctly without the need of a script when viewing the mode=manga_big&illust version.

So, what should we do for those images that were uploaded through master1200? Even if they weren't over 1200px long and/or high, their filesize still differs from the originals.

Should we upload the originals and make them the parent?

Updated

Reply
evazion

Ars said:

So, what should we do for those images that were uploaded through master1200? Even if they weren't over 1200px long and/or high, their filesize still differs from the originals.

Should we upload the originals and make them the parent?

I say upload and parent the original, tag the master1200 version as pixiv manga sample, and request that it be deleted in topic #8984. Manually deleting it instead of flagging it means everyone's favorites will be transferred to the parent.

Reply
Ars

evazion said:

I say upload and parent the original, tag the master1200 version as pixiv manga sample, and request that it be deleted in topic #8984. Manually deleting it instead of flagging it means everyone's favorites will be transferred to the parent.

Great, now if only source:*master1200.jpg or something similar would not cause a time out, it'd be much easier find them. There's also the possibility that the links were changed. Blah.

Reply
1 2 3 4 5