Retrieving Minimal Information

A place to share links to other websites and interesting articles
User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Retrieving Minimal Information

Post by euler » Thu Sep 27, 2012 1:42 pm

A new script has been made available to members to retrieve minimal information relating to the problems and account details. For obvious reasons it is not necessary to be signed in (an active session) for some of the calls.
  • https://projecteuler.net/minimal=problems
    This will return a new line for each active problem in the database containing the following information (delimited by ##):
    • id
    • description/title
    • date_published (timestamp)
    • date_last_updated (timestamp)
    • solved_by (number of members)
    If you are signed in then you will receive two additional values:
    • solved_flag (0=no / 1=yes)
    • answer (if solved_flag=1 otherwise empty)
  • https://projecteuler.net/minimal=problems;csv
    This provides the same information as above except it will force the browser to open/save as a comma separated file. The data will be delimited by commas and the description field will be qualified with quotation marks (actual quotation marks will be escaped).
  • https://projecteuler.net/minimal=X (where X is the problem id)
    This will return the raw problem content data in HTML.
  • https://projecteuler.net/minimal=profile (must be signed in to access)
    This will return details for the current account in a minimal format (delimited by ##).
    • username
    • alias
    • country
    • language
    • problems_solved (total number)
    • level
    • solved_string
    [The solved string is a "binary string" equal in length to the highest active problem and contains either 0 or 1 at a position corresponding to the problem id]
  • https://projecteuler.net/minimal=friends (must be signed in to access)
    This will return the same information as minimal=profile for your own profile and all those on your friends' list.
  • https://projecteuler.net/minimal=new
    Returns a list of the release dates of any new problems (one per line if more than one and delimited by ##):
    • id
    • release_date (timestamp)
    [If no new problems are currently scheduled then it will display the message, "No new problems currently scheduled"]
Image
impudens simia et macrologus profundus fabulae

sivakd
Posts: 217
Joined: Fri Jul 17, 2009 8:37 am
Location: California, USA
Contact:

Re: Retrieving Minimal Information

Post by sivakd » Sat Sep 29, 2012 2:46 am

euler, thanks for these. If I may suggest, why not generate csv or tsv files instead of html output with ## separators? csv/tsv files can be easily opened with Excel for analysis.
Image
puzzle is a euphemism for lack of clarity

User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Re: Retrieving Minimal Information

Post by euler » Sat Sep 29, 2012 5:34 pm

I've added support to retrieve the minimal information for problems as a csv file. See the main post for how to access this feature.

I used ## as a delimiter because not all languages have native support for comma separated files and as the description/title in the problems can contain commas it is not as simple as just using a string to array function with a comma delimiter. Also none of the minimal options specified text/html output, but because I was not specifying a header the output was being best interpreted as text/html by default. I've now changed the other minimal outputs to use text/plain.
Image
impudens simia et macrologus profundus fabulae

sivakd
Posts: 217
Joined: Fri Jul 17, 2009 8:37 am
Location: California, USA
Contact:

Re: Retrieving Minimal Information

Post by sivakd » Sat Sep 29, 2012 10:12 pm

euler, thanks for the csv support. I think there is no easy way to provide a pure text (stripping out the html tags and entities) header for csv. Also, in addition to the solved flag, providing the rank if available would be good.
Image
puzzle is a euphemism for lack of clarity

User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Re: Retrieving Minimal Information

Post by euler » Sat Sep 29, 2012 10:41 pm

What is the rank?
Image
impudens simia et macrologus profundus fabulae

sivakd
Posts: 217
Joined: Fri Jul 17, 2009 8:37 am
Location: California, USA
Contact:

Re: Retrieving Minimal Information

Post by sivakd » Sun Sep 30, 2012 3:55 am

Sorry, I meant the position in the fastest solvers table.
Image
puzzle is a euphemism for lack of clarity

User avatar
SoboLAN
Posts: 39
Joined: Sat Aug 18, 2012 2:21 pm

Re: Retrieving Minimal Information

Post by SoboLAN » Sun Sep 30, 2012 6:25 am

These pages are very useful for developers, thank you euler.

There is however a problem: introducing the CSV page is not a good idea because there are some problem titles that contain commas (","). CSV parsers can be easily confused by this. Using "##" is much better.

EDIT: I saw now that you already found this problem, you can ignore this post.
Image

User avatar
SoboLAN
Posts: 39
Joined: Sat Aug 18, 2012 2:21 pm

Re: Retrieving Minimal Information

Post by SoboLAN » Sun Sep 30, 2012 7:01 pm

Except for the problems 1, 143, 395 and 396, all problems have the same value in the "last_update" field: 18th September 2012. Why is this ? (I suspect why, I want to see if I'm right :P )
Image

Homez
Posts: 3
Joined: Mon Oct 08, 2012 9:02 pm

Re: Retrieving Minimal Information

Post by Homez » Mon Oct 08, 2012 9:22 pm

Hello to all folks!

I'm a Project Euler common solver (level 7) but I started thinking sometime ago of writing some framework to do keep some things more simple. This could be a JAVA SE application maintained in Eclipse. I could, for example, see the same list of prolems with their short descriptions, my solution status and the number of people having solved this problem. This now can all be taken from Project Euler site thanks yo yours scripts, euler! From me I could add there my actual solution times, so I could be eager to improve them. The solutions could run inside this app, so solution times could be maintained automatically in some config file or database like SQLite. It could use also some sorting and/or filtering so I could easily find problems solved by many people but which I haven't solved already by myself (I do it all now manually because I beleive that the problem I solve next is always among my unsolved problems that were solved by a larger quantity of members).

I could write all this just for my personal usage, but I could also share it to people that would like that (of course, without a single solution or common algorithm).

But I wonder: how can I login programmatically in order to get rid of these scripts? I know that I have to deal with session cookies in order to stay logged but I can't even login from my Java starting app :(

I saw this nice sample: http://ubuntuincident.wordpress.com/201 ... er-part-3/ Here some library browsers are used with Python. But I'm not ready to switch to Python, now I'm a Java guy :)
Image

Homez
Posts: 3
Joined: Mon Oct 08, 2012 9:02 pm

Re: Retrieving Minimal Information

Post by Homez » Mon Oct 08, 2012 10:10 pm

I can procceed! I've just added "login=true" to POST params - and now I see the list of problems, page 1. Now I can try to ged rid of session.
Image

Homez
Posts: 3
Joined: Mon Oct 08, 2012 9:02 pm

Re: Retrieving Minimal Information

Post by Homez » Tue Oct 09, 2012 1:08 pm

And I can't :( I try to set cookie that I got after posting a login form, for the next query. But I always get About page, being logged off. Can anyone help?
Image

Sjums
Posts: 3
Joined: Wed Nov 14, 2012 9:30 pm

Re: Retrieving Minimal Information

Post by Sjums » Wed Nov 14, 2012 9:51 pm

Hey euler, I'm all new (problem 8) but already hooked! Thanks for a nice riddle :-)

A simple request for the minimal data is; would it be possible to strip the HTML and serve it as plain text? I'm specifically thinking of the problem detail page:-)

Regards, sjums

User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Re: Retrieving Minimal Information

Post by euler » Wed Nov 14, 2012 10:09 pm

The problem content depends on HTML as it may contain images, superscript/subscript, or data contained in tables. If this formatting is removed then it would be impossible to interpret the problem correctly; for example, consider x = 32 without superscript: x = 32.
Image
impudens simia et macrologus profundus fabulae

User avatar
SoboLAN
Posts: 39
Joined: Sat Aug 18, 2012 2:21 pm

Re: Retrieving Minimal Information

Post by SoboLAN » Sat Dec 08, 2012 4:50 pm

Hi,

It makes sense to need an active session when going to http://projecteuler.net/minimal=problems or http://projecteuler.net/minimal=profile. The output seen in these cases depends on the user currently logged in.

But can you please remove this requirement for http://projecteuler.net/minimal=X (raw HTML of a problem) ? The output is always independent of the session, so it makes no sense in requiring it.

Programmatically logging in or automatically including cookies in the software written to retrieve the HTML is actually more difficult than I thought. It's a problem we shouldn't have...
Image

lucaswerkmeister
Posts: 3
Joined: Tue Jan 15, 2013 7:11 pm

Re: Retrieving Minimal Information

Post by lucaswerkmeister » Thu Feb 21, 2013 9:41 pm

SoboLAN wrote: can you please remove this requirement for http://projecteuler.net/minimal=X (raw HTML of a problem) ? The output is always independent of the session, so it makes no sense in requiring it.
Yes, that would be super cool... I just deployed a broken update to my plugin because of that.
Image

User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Re: Retrieving Minimal Information

Post by euler » Thu Feb 21, 2013 11:35 pm

Done.
Image
impudens simia et macrologus profundus fabulae

User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Re: Retrieving Minimal Information

Post by euler » Sat Mar 30, 2013 10:03 am

It is now possible to retrieve minimal information on the release schedule of new problems. See first post for details.
Image
impudens simia et macrologus profundus fabulae

User avatar
SoboLAN
Posts: 39
Joined: Sat Aug 18, 2012 2:21 pm

Re: Retrieving Minimal Information

Post by SoboLAN » Sun Oct 02, 2016 7:22 pm

@euler The "/minimal" links don't work anymore. Were they moved or renamed ? Can they be made available again ? Thanks.

PS: By "don't work" I mean they always return a HTTP 404 error.
Image

User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Re: Retrieving Minimal Information

Post by euler » Fri Oct 21, 2016 7:25 pm

Since I completely re-wrote all the scripts in 2014 this facility was not restored. I will look into it again and if it is viable under the new structure, and I get the time, I will re-write scripts to provide this functionality. I cannot give you even a hint of a time frame, but I will give it serious consideration.
Image
impudens simia et macrologus profundus fabulae

User avatar
euler
Administrator
Posts: 3047
Joined: Sun Mar 05, 2006 4:49 pm
Location: Cheshire, England
Contact:

Re: Retrieving Minimal Information

Post by euler » Wed Nov 02, 2016 5:44 pm

For those interested in knowing, the minimal information script has been re-implemented. I have updated the first post to reflect any changes that may have taken place.
Image
impudens simia et macrologus profundus fabulae

Post Reply