Difficulty Rating for New Problems

Announcements, comments, ideas, feedback, and "How do I... ?" questions
User avatar
hankinsohl
Posts: 12
Joined: Thu Nov 30, 2017 9:45 am

Difficulty Rating for New Problems

Post by hankinsohl » Thu Nov 30, 2017 9:55 am

New problems are presented without difficulty ratings.

Why so?

I suggest that new problems immediately have a difficulty rating - a tentative rating based on how difficult the author/publisher/whoever is involved in the publishing process thinks that the problem is.
=========
If there's an objection to the above proposal, along the lines of "we don't know how difficult this problem actually is given so few solvers" - well, go by the opinion of those as outlined above.

As people solve the problem - adjust the estimate - this can't be too difficult.
=========
TL/DR - Why is the difficulty rating absent for new problems? - this makes little sense.
Image

User avatar
hk
Administrator
Posts: 10563
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Difficulty Rating for New Problems

Post by hk » Thu Nov 30, 2017 12:30 pm

From the newsitems:
19 February 2015: Difficulty Ratings


After considerable work behind the scenes the team is pleased to announce the introduction of a difficulty rating system.

Difficulty ratings are based on the times taken to solve a problem since its publication date. For problems 277+ the data is taken from the fastest solvers tables and for older problems we are making use of data points taken from historical snapshots.

Due to the instability of the difficulty index with limited data it is not possible to provide a meaningful metric for newly released problems. Consequently we have separated the ten most recent problems (found under the new menu bar "Recent") from the rest of the problems (now called "Archives" in the menu). Recent problems will not have a difficulty rating and any problems in the Archives for which the fastest solvers table is not full will indicate that the rating is "not yet finalised".

Each problem in the Archives is given a difficulty rating ranging from 5% to 100% (in 5% increments). Within each of those twenty bands problems are then sorted by ID. Of course, no difficulty system is going to be perfect, and there are likely to be anomalies, but we hope that you find the system we have introduced provides a generally useful guide.

After a lot of discussions in the past we decided to keep it that way.
Image

User avatar
Animus
Administrator
Posts: 1800
Joined: Sat Aug 16, 2014 12:23 pm

Re: Difficulty Rating for New Problems

Post by Animus » Fri Dec 01, 2017 1:38 am

Judging the difficulty of a problem for the majority of users in advance is no easy task and the development team has often under- or overestimated it.
Nevertheless, as we have repeatetly said, we try to follow a regular publishing scheme by clustering the problems into batches of 6 problems each according to this pattern:

Medium - easy - medium - easy - medium - hard

When we happen to release two problems at the same time, we use the slot appropiate for the harder one. The next batch starts next sunday, so in order to get a rough orientation just keep track of the problems from then on.

Moreover, when just waiting a day or two, the number of solvers will give you quite a good idea where the problem difficulty is heading at.

LilStalker
Posts: 64
Joined: Thu Nov 03, 2016 4:32 pm

Re: Difficulty Rating for New Problems

Post by LilStalker » Fri Dec 01, 2017 8:41 am

I have been interested in that for a long time. Is it posible to get some examples where the development team highly overestimated the difficulty of the problem or the other case where it highly underestimated the difficulty of the problem?

I've heard that problem 257 was one of them, but I would be happy to know more examples.
Image

MuthuVeerappanR
Posts: 392
Joined: Sun Mar 22, 2015 2:30 pm
Location: India
Contact:

Re: Difficulty Rating for New Problems

Post by MuthuVeerappanR » Fri Dec 01, 2017 10:18 am

Given the caliber of the Dev team, I think overestimating the difficulty of a problem is a remote possibility. To 'overestimate' many members have to consider the problem difficult while it is actually easy. I doubt that.

Again for the same reason, 'underestimating' can occur more often. During some of the problem development discussions, I got the opinion that probability problems that are considered of medium difficulty ended up reaching max. difficulty levels (Problem 584 (View Problem) and Problem 499 (View Problem) may be two examples).

In any case, I too would be interested in some examples of under/overestimated problems.
Image
It is not knowledge, but the act of learning, not possession but the act of getting there, which grants the greatest enjoyment.

User avatar
hk
Administrator
Posts: 10563
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Difficulty Rating for New Problems

Post by hk » Fri Dec 01, 2017 11:35 am

LilStalker wrote:
Fri Dec 01, 2017 8:41 am

I've heard that problem 257 was one of them, but I would be happy to know more examples.
#257 was rated hard.
MuthuVeerappanR wrote:
Fri Dec 01, 2017 10:18 am
Again for the same reason, 'underestimating' can occur more often. During some of the problem development discussions, I got the opinion that probability problems that are considered of medium difficulty ended up reaching max. difficulty levels (Problem 584 (View Problem) and Problem 499 (View Problem) may be two examples).
#499 was rated Easy !
#584 was rated Medium
Image

User avatar
sjhillier
Administrator
Posts: 507
Joined: Sun Aug 17, 2014 3:59 pm
Location: Birmingham, UK
Contact:

Re: Difficulty Rating for New Problems

Post by sjhillier » Fri Dec 01, 2017 3:10 pm

Interesting discussion. I would like to suggest Problem 450 (View Problem) as another under-estimate. From it's position in the sequence, it must have been rated Medium, but turned out to be one of the hardest. I have a particular liking of this problem, since it was my first time in the top 100 (three months after its release). I have a suspicion that geometrical questions may be proving more challenging than anticipated. But that might just be my 'agenda' - I am under the impression that geometry isn't as well taught as it used to be. Alternatively, maybe it's the use of parametric equations that trips some people up - I don't remember many examples of that.

traxex
Posts: 77
Joined: Thu Oct 19, 2017 12:30 pm

Re: Difficulty Rating for New Problems

Post by traxex » Sat Dec 02, 2017 2:56 am

I think a certain hidden statistic plays a big role in some cases. While we know how many users have solved a problem, we don't know how many have tried to solve it, for let's say longer than 10 minutes. Most users have such a large number of unsolved problems that they can afford to be picky. In my case, the difficulty is not a very big factor when choosing which problem to attack next. I've ignored some straight-forward geometry problems and attempted some hard combinatoric ones.

Personally, I find combinatoric and number theoretic problem very easy to approach for several reasons:
  1. Small cases can be easily verified using pen and paper.
  2. There is often an obvious brute force approach that's easy to implement.
  3. The implementation can be done using only integer operations.
Problems in geometry or probability may share some of these features, but floating-point arithmetic in particular is one thing that I'm not a huge fan of. It's not fun to have a program that works for small cases and appears to be correct, but doesn't print the correct answer for the full problem. Often it's hard to be sure whether it's a problem of precision or some corner case that was missed; in integer problems, it's often easy to assume the latter, and with some care it's often possible to be virtually certain that the answer is correct before submitting.

Just as an experiment, I gave the "Easy" 499 a try. Its difficulty rating is very surprising, but on the other hand I probably would not have given it a serious try if it hadn't been mentioned on this thread. A significant portion of the time I needed to solve it was spent fighting with numerical precision. Ha ha.
Technically, everyone is full of himself.

User avatar
Animus
Administrator
Posts: 1800
Joined: Sat Aug 16, 2014 12:23 pm

Re: Difficulty Rating for New Problems

Post by Animus » Sat Dec 02, 2017 1:38 pm

hk wrote:
Fri Dec 01, 2017 11:35 am
MuthuVeerappanR wrote:
Fri Dec 01, 2017 10:18 am
Again for the same reason, 'underestimating' can occur more often. During some of the problem development discussions, I got the opinion that probability problems that are considered of medium difficulty ended up reaching max. difficulty levels (Problem 584 (View Problem) and Problem 499 (View Problem) may be two examples).
#499 was rated Easy !
#584 was rated Medium
So, in order to compensate for underestimating probability problems we were more careful and rated Problem 613 (View Problem) medium, but it turns out to be the easiest one this season so far. :wink:

philiplu
Posts: 20
Joined: Wed Aug 02, 2017 7:51 pm
Location: Redmond, WA, USA

Re: Difficulty Rating for New Problems

Post by philiplu » Sat Dec 02, 2017 9:34 pm

Hmm, I guess that means Poohsticks Marathon, Problem 589 (View Problem) was supposed to be easy, given the standard posting cadence? Well, that's discouraging :shock:. That one's been kicking my butt for a month now :). I could solve it if the one-minute rule was loosened to, say, a one-year rule.
Image

User avatar
sjhillier
Administrator
Posts: 507
Joined: Sun Aug 17, 2014 3:59 pm
Location: Birmingham, UK
Contact:

Re: Difficulty Rating for New Problems

Post by sjhillier » Sat Dec 02, 2017 9:54 pm

philiplu wrote:
Sat Dec 02, 2017 9:34 pm
Hmm, I guess that means Poohsticks Marathon, Problem 589 (View Problem) was supposed to be easy, given the standard posting cadence?
If it makes you feel better, I did assess this one as Medium, but was out-voted. It's one of those that depends on you having mastered a technique from earlier in the problem set.

User avatar
Animus
Administrator
Posts: 1800
Joined: Sat Aug 16, 2014 12:23 pm

Re: Difficulty Rating for New Problems

Post by Animus » Sat Dec 02, 2017 9:58 pm

With Poohsticks Marathon the dev team was divided between medium and easy.

Even for the team members being pretty well trained the subjective difficulty varies significantly. Most of us have felt stupid at same time not being able to find as good a solution (or any solution at all) for a problem that seemed to be easy for the other members.
On the other side, sometimes you're lucky and the first idea works right away, so even a challening problem suddenly seems to fall into place, something I hope all of you experience from time to time.

Note: Post war startet before sjhillier's.

philiplu
Posts: 20
Joined: Wed Aug 02, 2017 7:51 pm
Location: Redmond, WA, USA

Re: Difficulty Rating for New Problems

Post by philiplu » Sun Dec 03, 2017 1:30 am

sjhillier wrote:
Sat Dec 02, 2017 9:54 pm
philiplu wrote:
Sat Dec 02, 2017 9:34 pm
Hmm, I guess that means Poohsticks Marathon, Problem 589 (View Problem) was supposed to be easy, given the standard posting cadence?
If it makes you feel better, I did assess this one as Medium, but was out-voted. It's one of those that depends on you having mastered a technique from earlier in the problem set.
Eh, that's the danger of chasing awards. I decided in late June to skip to the last problem, 608, and get the On The Ball and One In A Hundred awards at the same time, even though I was only at 175 problems solved then. Then I decided to try for all the recent problem awards, until I only needed #589 to finish off the most recent 25. That works out fine for a lot of the number-theory problems, but my feel for probability problems is a lot shakier.

I've mostly decided to go back and work through earlier problems to avoid missing that incremental development (like the series of problems solvable as Pell's equations early on), but it's hard to avoid the temptation of trying out the latest problems. And, of course, just knowing that Poohsticks was meant to be easy triggers an idea or two I'll have to try out.

But to bring it back to the thread's topic, the fact that Poohsticks is at 90% difficulty, and took 28 weeks to get to 100 solvers, suggests the development team was a wee bit optimisitic on the perceived difficulty level. :) Seems I'm not alone in skipping around and missing the benefits of that incremental development.
Image

User avatar
hankinsohl
Posts: 12
Joined: Thu Nov 30, 2017 9:45 am

Re: Difficulty Rating for New Problems

Post by hankinsohl » Thu Dec 07, 2017 1:51 pm

Animus wrote:
Fri Dec 01, 2017 1:38 am
Judging the difficulty of a problem for the majority of users in advance is no easy task and the development team has often under- or overestimated it.
Nevertheless, as we have repeatetly said, we try to follow a regular publishing scheme by clustering the problems into batches of 6 problems each according to this pattern:

Medium - easy - medium - easy - medium - hard

When we happen to release two problems at the same time, we use the slot appropiate for the harder one. The next batch starts next sunday, so in order to get a rough orientation just keep track of the problems from then on.

Moreover, when just waiting a day or two, the number of solvers will give you quite a good idea where the problem difficulty is heading at.
Thanks Animus.

The mememh pattern is useful - as it allows me to judge the anticipated difficulty of new problems.

Out of curiosity - when was the pattern first implemented? Obviously the first 100 or so problems are all very easy - so the pattern cannot have existed since site inception.
======
Some observations:
First, some facts.
1) PE does not disclose the difficulty level of recent problems on the problem pages.
2) PE does, however, gauge the difficulty level of recent problems and publishes them in the repeated pattern mememh.
3) PE discloses this pattern here and likely elsewhere in the forums.

Conclusions:
1) The difficulty level of recent problems can be gauged by applying the mememh pattern.
2) Knowledge about the estimated difficulty of recent problems is limited to those who read this forum or have otherwise learned about the pattern.

Observation:
PE is reticent about disclosing the difficulty level of recent problems likely because estimates are often-enough wrong or very wrong. Nonetheless, PE actually does disclose the estimated difficulty level of recent problems in covert fashion - knowledge about the difficulty level of recent problems is limited to those who read this forum or otherwise learn of the pattern.

Anecdote:
The camel is a curious beast - able to travel without water for long distance; but of foul temperament - spitting at owners or biting them frequently. And discomforting to ride - with one or two large humps to contend with. One can only conclude that the camel is a desert horse designed by committee.

A Further Observation:
So too is the non-disclosure of estimated recent problem difficulty on the problem pages accompanied by tacit difficulty disclosure for PE forum readers - here is another camel.

Recommendation:
Disclose new problem difficulty estimates explicitly - easy, medium or hard is sufficient. This information is already disclosed tacitly. Correct such estimates periodically (1 or 2 days) if, given the estimated difficulty level, the rate of problem solvers is too high or low.

Moreover, explicit disclosure benefits site administrators who may need/want to deviate from the mememh pattern for various reasons.

Anti-Recommendation:
Continue to withhold recent problem difficulty estimates and dispose of the mememh pattern. Please don't do this.

Final Remarks:
I suppose that, ultimately, withholding of estimated problem difficulty is down to credibility - if often wrong or often very wrong - perhaps PE administrators fear that the difficulty estimate will be viewed as entirely without credence.

As remarked above, I strongly feel that this fear is entirely unfounded. Especially if the estimated difficulty level is concisely described as a "best-guess" - may well be wrong - in the hover text. And too, label recent problem difficulty information as estimated - as opposed to the archived problems where such estimates are much more firm.
Image

User avatar
hk
Administrator
Posts: 10563
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Difficulty Rating for New Problems

Post by hk » Thu Dec 07, 2017 2:24 pm

Something to ponder about what this has to do with your previous post:
Did you ever hear of a self fulfilling prophesy?

Oh and by the way: I don't like fairy tales about dromedaries either,
Image

User avatar
hankinsohl
Posts: 12
Joined: Thu Nov 30, 2017 9:45 am

Re: Difficulty Rating for New Problems

Post by hankinsohl » Thu Dec 07, 2017 2:31 pm

hk wrote:
Thu Dec 07, 2017 2:24 pm
Something to ponder about what this has to do with your previous post:
Did you ever hear of a self fulfilling prophesy?

Oh and by the way: I don't like fairy tales about dromedaries either,
Something for you to ponder:

Do you ever dismiss others out-of-hand, without regard for what they say or write? Perhaps piling unwarranted insults into the mix?

One last observation - you, hk, are usually persuasive and elegant, in your posts and your problem summaries in the forums.

And yet, there's this:

"Did you ever hear of a self fulfilling prophesy?"

Rhetorical I suppose - but also very insulting. Think about it if you don't get this.

And there's also this:

"Oh and by the way: I don't like fairy tales about dromedaries either,"

You've characterized my analogy about camels and non-disclosure of problem difficulty as "fairy tales" - surely this is off-the-cuff as it's below the otherwise excellent literacy level of other posts you've made. God only knows how suddenly analogies are fairy-tales.

And then, there's the "," instead of a "." - obviously you've knee-jerked into an emotional response, as opposed to actually giving my post some thought.
Last edited by hankinsohl on Thu Dec 07, 2017 2:45 pm, edited 1 time in total.
Image

User avatar
hk
Administrator
Posts: 10563
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Difficulty Rating for New Problems

Post by hk » Thu Dec 07, 2017 2:44 pm

Dear Hankinsohl,
despite the fact that we took a lot of time to answer your previous question it seems that you insist on requesting that the team posts the estimated difficulty beforehand (or keeps it adjusting on the fly).
As I already guessed you didn't want to reflect the ultimate reason why this shouldn't be done.
If we would solvers might think: "hey that's a hard one, I don't even start with it" or "hey, that one should be easy and become frustrated it turns out to be that easy".

Thanks for not taken us serious enough to think up this yourself when invited to do so and starting to tell us a lot about camels.
Image

User avatar
hankinsohl
Posts: 12
Joined: Thu Nov 30, 2017 9:45 am

Re: Difficulty Rating for New Problems

Post by hankinsohl » Thu Dec 07, 2017 2:46 pm

hk wrote:
Thu Dec 07, 2017 2:44 pm
Dear Hankinsohl,
despite the fact that we took a lot of time to answer your previous question it seems that you insist on requesting that the team posts the estimated difficulty beforehand (or keeps it adjusting on the fly).
As I already guessed you didn't want to reflect the ultimate reason why this shouldn't be done.
If we would solvers might think: "hey that's a hard one, I don't even start with it" or "hey, that one should be easy and become frustrated it turns out to be that easy".

Thanks for not taken us serious enough to think up this yourself when invited to do so ans starting to tell a lot about camels.
Actually, dear hk, did you read my previous post?

Another "fairy tale" for you - since you seem to think in these terms - catch more bees with honey.
==========
>> If we would solvers might think: "hey that's a hard one, I don't even start with it" or "hey, that one should be easy and become frustrated it turns out to be that easy".
Actually I'm fully aware of this reasoning and have clearly pointed out why I think that these fears are completely unfounded. But you'd have had to actually read and thought about my post to supply reasonable counter-arguments.
Last edited by hankinsohl on Thu Dec 07, 2017 2:52 pm, edited 1 time in total.
Image

User avatar
hk
Administrator
Posts: 10563
Joined: Sun Mar 26, 2006 9:34 am
Location: Haren, Netherlands

Re: Difficulty Rating for New Problems

Post by hk » Thu Dec 07, 2017 2:48 pm

Yes I did.
Image

User avatar
hankinsohl
Posts: 12
Joined: Thu Nov 30, 2017 9:45 am

Re: Difficulty Rating for New Problems

Post by hankinsohl » Thu Dec 07, 2017 2:53 pm

hk wrote:
Thu Dec 07, 2017 2:48 pm
Yes I did.
Strange - your posts indicate otherwise.

Here's a thought - two reasonable people may disagree.

In my view, courtesy and mutual respect dictate that a simple acknowledgement of disagreement suffice.
Image

Post Reply