New Ranking System (Feedback)

[tklalmighty]

Also, maybe the scores page should show people within 500-1000 rank (the closest 1000 ranks) of the viewer? This way, newcomers will not feel put off by being "off the radar."

@neverforget: It may just be easier to go back to ranking the top 1000, and then show each person not in the top 1000 what percentile they are currently in.

What if you made certain problems worth more "points" (just going towards your level) than others, based on difficulty or the number of people who solved them. That way if you solve lots of "hard" problems you could jump up in your level more... or if for some reason thousands of people solved what was thought to be the "hardest" problem, your ranking would change accordingly (aka drop!).

@greekcandyman: LOL see my proposal below. (By the way, what you describe was the old system, if my mind serves me correctly.)

Oh, and I'm not in the 90% boat either.. I only solved 37! But then again, I havent graduated from high school yet!
(I think you should add an "age" thing to profile while you're at it! )

@greekcandyman: Age sounds like a great idea, as does profession. Like you, I am not yet out of high school, and I think both age and profession would give all of us a better idea of where we all come from.

IMO, most of these ideas regarding really seem to contribute a lot of unnecessary clutter to what is already a somewhat elegant system. I can personally tolerate most ranking systems you can think of (including military), but I don't think the names are really necessary. I think that the numbers are good enough.

I feel the only major changes that need to be made are to how problems are ranked by "difficulty". I would like to see a revival of the old system, ranking people by point values attributed to each problem. Although this time, point values wouldn't be attributed directly for the number of people who solved the problem, but for a ratio determined by the number of solvers and the length of time since the problem was released.

In other words, each problem would be assigned the point value P=FLOOR(S/R), where S is a scalar (I recommend 100), and R is the ratio of the number of solvers to the number of days since the problem was released.

A couple of examples:

• On 9 July 2008, problem 1 was the easiest problem, with a ratio of 24884 solvers/2469 days = 10.0786 per day. If the scalar is 100, then problem 1 would have been worth 9 points by that value system. On 29 July 2008, problem 1 was still the easiest problem with a ratio of 25953/2489 = 10.4271 per day. It would still be worth 9 points.
• On 9 July 2008, problem 88 was the hardest problem, with a ratio of 485/1251 = 0.3877, being worth 257 points. On 29 July 2008, it was the second hardest problem, with a ratio of 512/1271 = 0.4028, and it was worth 248 points.
• The major problem with this system deals with the newer problems. On 9 July 2008, problems 201 and 202 had been out for 4 days and the ratios for each were 8.75 and 9.5, respectively. The problems would have been the 198th and 200th easiest problems respectively, worth 11 and 10 points. Only time gives a better gauge of the relative strengths of each problem, as on 29 July 2008, these problems had become the 169th and 182nd hardest problems respectively, worth 33 and 25 points. (The ratios had dropped to about 2.9583 and 3.9167, respectively.)

On 9 July 2008, the maximum number of points one could have would have been 19639, and on 29 July 2008 that would have been 19688.

There are now multiple ways of ranking people:

• Unweighted: the current system, ranking people by how many problems each have solved
• Recent: the new current system, showing, for each person, how many of the most recent 100 problems have been solved
• Weighted: my proposal, similar to the old system, assigning point values to problems and ranking people by that

I feel the weighted system should be used/added because, no offense to anyone, one should be able to say that solving problems 88, 106, and 143 represent a much greater "feat" than solving problems 1, 2, and 6. It also provides a much better gauge of how easy or difficult problems are with respect to each other. Problems like 145 are really made out to be harder than they really are, only because they haven't been out for as long and solved by as many people.

Perhaps the level idea could be adapted to the weighted points system: new level every 500 points?

What do you guys think?
-tklalmighty

P.S. I have full data from 9 July 2008 00:00 GMT and 29 July 2008 04:00 GMT which I can post if anyone wants to. I'll be taking more data over the next few weeks and I can post that eventually as well.

[stalepretzel]

I just realized that the genius percentage was gone (wow I'm slow...)
I really liked it. The thing was, you didn't move up in percentage EVERY TIME. That gave me a little extra push to continue solving (well, I can't have solved a problem, but have no percentage to show for it! Better "level up" again!) I really like seeing the genius percentage slowly increase .

Meanwhile, I've solved 63 problems. (I would have just achieved 31% genius!) So I guess I'm still relatively low on the scale. But, I must say, I still remember what it was like to be completely new (a couple months ago). I was completely unoffended by the scoring system telling me that I wasn't a genius at all. I think it's an inspiration. I doubt any reasonable person could be offended when they join a site and are told that they haven't done anything yet!

In other word, "Don't mind the noobs....."

[tklalmighty]

But, I must say, I still remember what it was like to be completely new (a couple months ago). I was completely unoffended by the scoring system telling me that I wasn't a genius at all. I think it's an inspiration. I doubt any reasonable person could be offended when they join a site and are told that they haven't done anything yet!

In other word, "Don't mind the noobs....."

I think you make a good point; when I first started it took solving a few problems to even get a genius rating (this was back when problems had point values, and all the scoreboard included all users). I think I was more "offended" (if I can use that term there) by the idea that it would take me time to rise in the rankings than by the fact I hadn't solved very many problems, and I must say, it gave me inspiration as well to do more problems.

[neverforget]

But, I must say, I still remember what it was like to be completely new (a couple months ago). I was completely unoffended by the scoring system telling me that I wasn't a genius at all. I think it's an inspiration. I doubt any reasonable person could be offended when they join a site and are told that they haven't done anything yet!

In other word, "Don't mind the noobs....."

I think you make a good point; when I first started it took solving a few problems to even get a genius rating (this was back when problems had point values, and all the scoreboard included all users). I think I was more "offended" (if I can use that term there) by the idea that it would take me time to rise in the rankings than by the fact I hadn't solved very many problems, and I must say, it gave me inspiration as well to do more problems.

Yes, this is a good point to make. I must also say that being told I was 2% genius made me try to find something (quickly) to remedy the statement (even if it was the easiest 20 problems listed).

It may just be easier to go back to ranking the top 1000, and then show each person not in the top 1000 what percentile they are currently in.

I still think it might be better to show the status of neighbors as well, to give people a better idea of where they stand, and where they should move next. As Tirian puts it:

...I barely care about the progress of anyone else. (The one exception is that I like looking at the problems solved by people near my "rung" of the ladder to get a sense for the problems that might be easiest.)

Another way I used to determined problem order was to sort unsolved problems by ascending difficulty.

In terms of "campaigns" or "achievements," I think people wouldn't like to have awards, titles, and medals removed once they achieve them. They can still motivate people to work hard without throwing them off the boat or out of the club. We can have a trophy for solving the first 20 problems, another trophy for solving the first 50 problems, then 100 problems, then 200 problems, etc. If we give an award for completing the LAST 20 problems, they will have to maintain it every update. This might add some challenge. Maybe we can have both kinds of trophies, though if we are considering the newcomers, they will likely finish the first twenty problems before they finish the last twenty problems.

[euler]

These suggestions and discussions have been quite excellent and I think that it is fair to say that there has been a general convergence of ideas, which is why I think we're going to have to draw the line. I propose we move forward with the following "structure"...

We use numeric levels: Level 1 (25-49), Level 2 (50-99), Level 3 (100-199), Level 4 (200-399), Level 5 (400+). I think it is fair to say that for anyone having solved in excess of 400 problems the they would be more than happy to know they have achieved the maximum level and competing with other "level 5" members would be their priority. Rather conveniently this allows the use of the five Platonic solids to be assigned. Not that they bear much significance other than providing distinct groups.

Each of these groups would have separate score tables based on the number of problems solved, but in addition we would have a separate "top scorers" table which utilises a slightly more dynamic method; I also think it would be nice to define the "top scorers" as "Eulerians". I still think that measuring on the most recent, say, 25 problems works rather well, as it provides a realistic target for talented newer members who want to compete with the best of the best, after six months of inactivity you would drop of the table, and it provides an alternative competition to the long term level advancement (which can never be taken away from you).

At the end of the day, people do Project Euler because they enjoy solving problems and I believe that this proposed system keeps problem solving at the forefront, whilst introducing a simple, yet unobtrusive "reward" system.

Thoughts?

[elendiastarman]

What about Level 0?

[LarryC]

I genuinely like it. But I'm easy and liked your military theme too.

As for the ranking of problems' difficulties, I would prefer that each remained 1 point so to speak. I don't think there's an accurate way to rate them and can't see much benefit to it. That said, I'm easy on this too!

[elendiastarman]

What about Level 0?
Also, how about we put the different systems to a vote?

[hk]

We use numeric levels: Level 1 (25-49), Level 2 (50-99), Level 3 (100-199), Level 4 (200-399), Level 5 (400+). I think it is fair to say that for anyone having solved in excess of 400 problems the they would be more than happy to know they have achieved the maximum level and competing with other "level 5" members would be their priority. Rather conveniently this allows the use of the five Platonic solids to be assigned. Not that they bear much significance other than providing distinct groups.

Wow, level 5 (400+) would take about 4 years from now at a rate of 50 problems a year, and level 4 would take another two years from now.
I'm sure the "dev team" does its best develeoping problems, but increasing the rate to two or three problems a week would exhaust both the dev team and the solvers.
So perhaps we should introduce levels at each 50 and not with steps of 100.

[neverforget]

What about Level 0?

I would suppose they are unranked. That is, there is not considered any significant achievement until 25 problems are reached.

As for the ranking of problems' difficulties, I would prefer that each remained 1 point so to speak. I don't think there's an accurate way to rate them and can't see much benefit to it.

The first ranking I saw was to have difficulty be inversely proportional to number of people who solve it. So to speak, there is an accurate way to measure difficulty, but I also do not see much benefit to this method. Such a system makes it hard for newcomers to gain ground, as they take out the easy problems to get a grip on Project Euler.

I like the 5-level platonic solids system Euler suggested and ranking based on the last 25 problems. Keeping different score tables seems to be the best way to address the issue of keeping both newcomers and veterans satisfied with their performance. For the solids, I think it would be sufficient to use tetrahedron, cube, octahedron, dodecahedron, icosahedron, and perhaps, if like hk mentioned, we have finer divisions, we can have the soccer ball polyhedron (truncated icosi?) as well, or even the sphere for 400+.

[tklalmighty]

As for the ranking of problems' difficulties, I would prefer that each remained 1 point so to speak. I don't think there's an accurate way to rate them and can't see much benefit to it.

The first ranking I saw was to have difficulty be inversely proportional to number of people who solve it. So to speak, there is an accurate way to measure difficulty, but I also do not see much benefit to this method. Such a system makes it hard for newcomers to gain ground, as they take out the easy problems to get a grip on Project Euler.

I think I'm being overly repetitive (pardon me if I am), but the one thing I see wrong with that is that people have had a much longer time to solve earlier problems than more recent problems. That is why each of the difficulty methods I've seen on Project Euler are so extremely weighted toward the earlier problems. Say one hypothetical person solves only the 25 "easiest" problems and another solves only the 25 "hardest" problems. Does it do either of them justice to say these two are equal by making each problem worth one point?

IMO difficulty should be inversely proportional to the average number of people who solve it, and personal scores should be based on that, not how many problems one has solved. (That is basically the gist of my earlier post.)

As for having numeric levels, I'm okay with that, although Level 5 does seem a bit unapproachable, wouldn't you say? I propose levels every 20 or 25. It is a much more attainable task for anyone, and the ranges for Level 1 (and the unranked level) can be equal to that of all the other levels. I would rather not assign names to levels, but that is only my opinion.

Or how about this? We could make Project Euler extremely unique and have levels for every 37 questions solved. ...or maybe not...

[neverforget]

Say one hypothetical person solves only the 25 "easiest" problems and another solves only the 25 "hardest" problems. Does it do either of them justice to say these two are equal by making each problem worth one point?

Yes. According to hypothesis, second person can still solve the 25 "easiest" problems, and the first can still attempt the 25 "hardest" problems.

On a separate point about "difficulty," keep in mind what is easy to one person may be a conundrum to another. People without Computer Science background may find dynamic programming foreign, while coders without much mathematical background may find concepts such as continued fractions or Diophantine equations equally slippery. No two people are the same.

[greekcandyman]

I think I'm being overly repetitive (pardon me if I am), but the one thing I see wrong with that is that people have had a much longer time to solve earlier problems than more recent problems. That is why each of the difficulty methods I've seen on Project Euler are so extremely weighted toward the earlier problems. Say one hypothetical person solves only the 25 "easiest" problems and another solves only the 25 "hardest" problems. Does it do either of them justice to say these two are equal by making each problem worth one point?

IMO difficulty should be inversely proportional to the average number of people who solve it, and personal scores should be based on that, not how many problems one has solved. (That is basically the gist of my earlier post.)

As for having numeric levels, I'm okay with that, although Level 5 does seem a bit unapproachable, wouldn't you say? I propose levels every 20 or 25. It is a much more attainable task for anyone, and the ranges for Level 1 (and the unranked level) can be equal to that of all the other levels. I would rather not assign names to levels, but that is only my opinion.

Yes, I like this! Having levels so spaced apart makes it boring while you are striving to reach another level..
And you should bring the Genius Rating back! My brain feels sad now that nobody is calling it a genius!

[alexk]

I like the weighted approach that tklalmighty proposed. Even if it will not end up being used for the ranks, it's still the most objective method to order problems by difficulty and can be useful when picking up the next problem to solve.

As for the levels, why fix the number of problems to be solved? If there N problems available (N=202 as of today) and we want M levels (M=5?) then the i-th level is reached when int(N*i/M) problems are solved. That will force everyone to keep on solving the problems

[euler]

Sorry if it seems like I am ignoring people; it's not my intention. The problem with dynamic scoring is the demands placed on the server. Some older members have already mentioned that we used to utilise a clever scoring system for which the value of a problem was inversely proportional to the number of members that had solved it, but much to our dismay it was necessary to abandon it. To use current figures... we have close to 30000 active members and after calculating values for the current set of 202 problems it would require over twelve million calculations before performing a numeric sort on an array containing 30000 elements each time the score table needed to be updated. As the problem set continues to grow and the membership increases the demands literally multiply.

Another suggestion was about the most improved member. However, I know it would only be a small number guilty of this, but a less honest person could solve the bulk of problems under a dummy account then enter the answers in a short space of time on their real account giving the impression that they solved them quickly.

It has also been pointed out that although some problems are deemed to be easier, difficulty is relative to knowledge and experience. It isn't as if more experienced members are only allowed to solve hard problems and vice versa. Quite simply, everyone can solve easy problems but only the more experienced will be able to go on to the harder ones, thereby creating an "accurate" measure of experience.

As neverforget explained, it was intentional to not award a level until a member had solved at least 25 problems. This was partly for the practical reason of reducing the size of the cached files for each separate level score table; currently just under 80% of members never actually solve 25 problems, meaning that only 6080 members would need to be cached in the tables.

If the previously proposed boundaries were implemented then we would start with around 80 level 4 members, but in light of hk's point about the unreasonable gap before achieving level 5, how about the following... ?

Level 1, Tetrahedron, 50-99
Level 2, Cube, 100-149
Level 3, Octahedron, 150-199
Level 4, Dodecahedron, 200-249
Level 5, Icosahedron, 250-299
Level 6, Sphere, 300+

This would mean that our current top scorers could advance to level 5 by this time next year, and would still have one more level to advance to the following year.

Currently just over 20% of members have solved 25 problems or more, whereas around 10% (2817) members have solved 50 or more. Raising the first level to 50 problems means that less people will appear in the score tables, but bear in mind that the previous system only recorded the top 1000, plus this was always a moveable target as the number of problems increased. I suspect that if members knew that solving 50 problems would guarantee them a place in one of the tables it might provide the incentive to continue.

[edit]I've just seen alexk's suggestion about having score boundaries equally divided between the size of the current problem set... nice idea.[/edit]

[tklalmighty]

Sorry if it seems like I am ignoring people; it's not my intention. The problem with dynamic scoring is the demands placed on the server. Some older members have already mentioned that we used to utilise a clever scoring system for which the value of a problem was inversely proportional to the number of members that had solved it, but much to our dismay it was necessary to abandon it. To use current figures... we have close to 30000 active members and after calculating values for the current set of 202 problems it would require over twelve million calculations before performing a numeric sort on an array containing 30000 elements each time the score table needed to be updated. As the problem set continues to grow and the membership increases the demands literally multiply.

No, I understand and remember fully well why it was necessary to abandon the old method. I just feel a weighted system would make for a better discernment over which problems people found to be "easier" or "more difficult." Quite frankly, we already keep track about how many people solve each problem, and each problem has a date tag on it, so being able to sort the problems by this ratio of solvers to time should not require that much more. (I see the memory and storage problem should lie when we start attributing weighted points to people.)

Maybe I should have been a bit more clear in my wording. I think what I'm trying to describe as "easy" or "difficult" is based on how many people have been able to solve it. Looking at the fact that 5 times as many people have solved 145 than have solved 143, it appears that more people have found 145 quicker to get than 143. Ironically, I point out, I got 143 in much less time than 145. The overall consensus appears to have deemed 145 "easier", even though there is a minority of people (like me) who view 143 to be a bit easier than 145. That overall consensus seems to be the best fit for what the difficulty rating is looking for -- someone looking to find an "easier" problem will look to the difficulty rating, and if more people are able to solve it each day, it stands to be a problem more worth looking at, even though it may not be the "easiest" problem to the specific person.

P.S. Has the suggestion ever come out of categorizing the problems...number theory, geometry, etc.? Or is it that some of the problems cross over into too many of the categories that the categories become obsolete?

[JohnMorris]

Sorry if it seems like I am ignoring people; it's not my intention.

On the contrary - you have been extremely responsive, clearly thinking about what people have been saying.

The problem with dynamic scoring is the demands placed on the server. Some older members have already mentioned that we used to utilise a clever scoring system for which the value of a problem was inversely proportional to the number of members that had solved it, but much to our dismay it was necessary to abandon it. To use current figures... we have close to 30000 active members and after calculating values for the current set of 202 problems it would require over twelve million calculations before performing a numeric sort on an array containing 30000 elements each time the score table needed to be updated. As the problem set continues to grow and the membership increases the demands literally multiply.

I'm neither for nor against scoring according to estimated problem difficulty. Whatever you go with is fine by me (I'm easily satisfied). But I am curious about why this should be so hard: this does not seem like a huge task to me, now that the table is only updated once an hour. 12 million calculations and a 30,000 element sort? That's sounds like "a few seconds by brute force" if it were a PE problem. Heck, the hard part of the calculation - calculating the worth of each problem - could even be reduced to once a day.

Or am I missing something?

That's a bit of a thread hi-jack, so although I am curious to know the answer, I'll understand if you don't want to go into it here.

If the previously proposed boundaries were implemented then we would start with around 80 level 4 members, but in light of hk's point about the unreasonable gap before achieving level 5, how about the following... ?

Level 1, Tetrahedron, 50-99
Level 2, Cube, 100-149
Level 3, Octahedron, 150-199
Level 4, Dodecahedron, 200-249
Level 5, Icosahedron, 250-299
Level 6, Sphere, 300+

I think those bounds may be a little too wide, just from my own experience while climbing the ranks.

I joined PE about a year ago and reached 100% for the first time something like 6 months later. At that time the scores table was huge, including everyone who had solved even a single problem. After getting half way up the table (about 25 problems, I think it was), my personal goal became always to get up to the next higher page in the listings. And although that got harder and harder as time went on, it always seemed to be in reach. "I've made page 7, now just ten problems to solve and I'll make page 6". And so on, up the table. Reaching page 1 was a huge high point for me. Reaching 100% after that was almost an anti-climax.

So for me the score spread between one page and the next the (very) old table made effective incentive points. But maybe that was just me

[Tommy137]

P.S. Has the suggestion ever come out of categorizing the problems...number theory, geometry, etc.? Or is it that some of the problems cross over into too many of the categories that the categories become obsolete?

I had a similar idea this night. For a person that is completely new to PE it takes a lot time to read through and completely understand (especially if you aren't a native speaker) all 200+ problems. Maybe this person doesn't have a vast knowledge of mathematics but knows a bit about graph theory, he'd probably like to see if there are any GT related problems. So if we use one or two(for cross-overs) tags for each problem and get a sort of overview od all problems that are tagged with e.g. "graph theory", one would fast find problems that one may easily solve and after that try some other categories.

Of course the tags should be really vague so that there aren't any hints as how to solve a certain problem.


(There could be even tables for each category so new members could compete with old members in a certain category)


Just an idea...

[hk]

The problem with tagging the probems is that either the tags are too vague to be helpful or are too revealing.
Even a tag could steer someone away from another equally valid approach.
Someone that is new to PE does best to sort the problems on "difficulty" and solve the problems that have been solved by the most users and read through the pdf's and the first pages of the forum.
The way PE has grown that will guarantee a short course in necessary techniques.

If we decide on 6 levels (which is fine by me) I don't think it useful to have one or more levels that exceed the problem set.
Relative boundaries seem a good idea to me.
How about the following levels:
Level 1, Tetrahedron, 50-99 problems solved.
Level 2, Cube, 100 or more but less than 70% of the probems solved. (currently =100-140 solved)
Level 3, Octahedron, 70%-80% solved
Level 4, Dodecahedron, 80%-90%
Level 5, Icosahedron, 90%-99%
Level 6, Sphere, >=99% solved.

Those gaps can be quite wide but then there will be a lot of persons in the same category so the listings will be paginated anyhow. Thus members can put their own subgoals using the pages.
To prevent too long a list for Level 1 we could require that to be listed in Level 1 one should be acitve solving problems:
let's say has solved a problem within the last 60 days. If one has solved more than 100 problems that is an achievement that does not justify being not listed because one did not solve a problem recently.

[ed_r]

Another suggestion was about the most improved member. However, I know it would only be a small number guilty of this, but a less honest person could solve the bulk of problems under a dummy account then enter the answers in a short space of time on their real account giving the impression that they solved them quickly.

So what if a less honest person does that? Improvement scores are calculated anew each week, so the cheater would get only fleeting benefit. And it's not as if the existing PE system isn't open to more permanently-effective abuse already, as I'm sure you're aware.

When viewing scores, it should be possible to select on three different types of score (total, last-25 and improvement) and - as at present - filter further by country/language etc.

Whether or not you go to the further fuss of creating leagues, ranks, castes or whatever is neither here nor there for me.