Cool.

Source?

@CozyPlanes already on the 'hub.

It's incredibly incomplete. Only profiles are complete right now. It's under my cursesRant project. I need a rant API for it (obviously)

@CozyPlanes oh, and don't expect comments. I don't fly like that on personal projects until the end.

Hey @dfox... Friend.... When do I determine when to reset the "skip" argument in the API to zero? Is it after a certain amount of time...?

@AlgoRythm very cool!

The skip parameter for which endpoint? For most of them it is just for paging.

@dfox So for example, I download 50 rants from the Algo. Then I do it again. If I don't set the skip parameter to 50, I get the same rants as the first time. That's fine, but the next day, I can download 50 different rants with the skip parameter at 0 because time had passed.

When do I know when it's okay to set back to 0?

@AlgoRythm for algo it is using our algo v2 that has no skip paean. Here’s info on algo v2 I wrote up for someone else in the past:

“The best way to think about it is there's no more concept of "skip" with it. That's the main difference. You make the same request but there's a small difference. When you make a request to the rants endpoint with sort algo, you'll notice that it sends back a key called "set". That value is the unique id of the returned rants, specific to the logged in user.

The idea is when you request the rants from algo, you store that set id in memory. When you go to fetch the next series of rants (ex. when user requests next page, scrolls to bottom on infinite scroll, etc.), you send the param "prev_set".

prev_set does the following: it tells the algo what the previous set you just finished looking at was (the set id of that returned set) so the server will: not return any rants from that set (since algo v2 doesn't show rants you've already seen until you run out of algo rants), and it will also mark all of the rants in that set id as seen on the server-side so you don't have to keep track of them.

One more thing, and this part is optional: the process the server uses to mark sets of rants as read (based on prev_set) is asynchronous. That means that you very quickly try to load algo v2 rants, it's possible you could see the same ones if the server hasn't yet processed a previous prev_set as "seen". The way to prevent this, if it's something you're concerned about, is to send multiple set ids in "prev_set". Here's how that works. You can send "prev_set" as a comma delimited list of previous set ids. Ex. "12345,6789,3456". The first id in the list always needs to be the previous set that you're marking as read. The other ones can be any ones that were previously returned. They get passed to the algo so even if the "seen" rants from a previous request haven't been written yet, it knows not to show any rants from those previous set ids. This part is more just for safety, and might not be needed.

Lastly, algo v2 is only available for logged in users (requests with user/token). That might be why it wasn't reported on your app, since logged out users never would have seen the issue since they automatically get algo v1. So for logged out users, you should still send the skip param (you can send it for both, v2 just ignores it.)”

@dfox That's incredibly interesting. I thought "set" referred to a specific feed of rants that the Algo was serving, and you kept iterating the skip argument until the set variable changed. I was quickly confused when the set variable returned different values each time!

What do you recommend for logged out users? Just a paging system like on the website? A specific amount of time? I know there are quite a few anonymous lurkers who would enjoy a proper experience.

@AlgoRythm every other rant sort (Ike top, recent) used skip to page the same way logged out/logged in. Logged out algo uses the old algo and paging works with skip like the other endpoints.

@dfox are user IDs stored sequentially? For example, I'm user ID 34534 (I'm not, that's random). Is the next person to register given the ID 34535?

@AlgoRythm i think so

@CozyPlanes By my testing, either a LOT of people delete their accounts, or they aren't

@CozyPlanes Just hit ID # 374602 and only three other profiles were found. Mine is 374269 (That's where I started). I scanned 300+ IDs.

@AlgoRythm they are not sequential, ids are shared between rants, comments, users, etc.

@dfox Shit. Is there any way to sequentially grab all of them...?

@AlgoRythm there is not. In general we discourage people from trying to iterate through the entire userbase and it’s not really something we want the API being used for since it uses a lot of resources on our end and the general use case can lead to data-mining related things that we don’t really encourage either.

@dfox But... I love data mining... I mean science. I said science.

Oh well. I already wrote the Python script and ran it for a few minutes, so I guess I had my fun. 300 scans isn't too bad...

@AlgoRythm you’re welcome to use it for your personal use :) I more just mean there’s no way to sequentially get users and there’s no plan to add it, since the only API endpoints we offer are the ones our apps use and we wouldn’t need that kind of functionality for anything.