cache rating - request for comments [closed]

[rediguana]

Bump

We've just been having a big discussion on ratings in the Eastern Isles and I thought I'd like to share the link, as well as some feedback on the gc.com.au recommendation system.

So, who's the most addicted? (starts slow, soon moves to rating)
http://www.gps.org.nz/forums/viewtopic.php?t=1414

Anyway, I think its great to having the recommendation feature on gc.com.au. Good on you for implementing it!

As a result of the discussion thread above, I've come up with some suggestions and issues with the current system.

1. 10 Recommendations is too few to be useful as I want to recommend more than 10 out of the 350-odd I've found in New Zealand. Yet someone that has found 20 caches can recommend as many as me? I think there needs to be a proportionate amount provided based on finds. Whether its 1-in-10 or whatever else, we need to be able to draw upon the experienced cachers to recommend caches for newer cachers.

2. 1 Recommendation of a cache should not turn it gold. There needs to be a minimum threshold of recommendations, e.g. 3 or 5 before it becomes listed as gold. One person recommending a cache does not make it worthwhile. A certain minimum of people should be required to recommend the cache before it is listed as gold. Three would seem to me a minimum number. I'd be much more trusting of a recommendation with 3 different individuals over 1. This would help counter the increased number of recommendations made by step one. The threshold may be set so that it identifies only the top x% of caches, and may need to be set higher every few months as more recommendations are made to ensure that there is always only a certain % i.e. top 5% shown.

3. It should still be possible to browse an individual members recommendations, and it would be very cool to have the website identify cachers that have made recommendations similar to you. You could then use their indications as a guide.

Hope I'm not stepping on any sacred ground here, but I'd much rather see improvements made to gc.com.au, than try and do our own ratings in NZ.

Cheers Gav

[rediguana]

Update...

I believe that the problems of having too many caches identified as recommended can be solved by raising the threshold for being displayed.

Basically rank the caches by recommendation total, and take the top x% and show those as gold. This may need to be done on a state by state basis to allow for different thresholds for states with different levels of geocaching activity.

Cheers Gav

[Mind Socket]

All good suggestions Gav, I'd be interested to hear other people's thoughts on this also. Wearing the cacher and developer hat here ...

think there needs to be a proportionate amount provided based on finds

I agree. I've clicked on a cache that appears recommended by a cacher with only 10 finds. For that reason, I'm less inclned to think it's an informed recommendation but it would be nice to reduce the number of these with some sort of ratio. Just as 10 is an arbitrary number now, picking an appropriate ratio will be just as arbitrarty. Maybe it could be done such that the average number of finds per cacher would allow for 10 recommendations.

1 Recommendation of a cache should not turn it gold

Again I agree for the same reason, but I'm not sure that it shouldn't be recognised at all. There is the issue of how often a cache is visited. Eg "4 day hike required cache" might only get 1 visit a year, so it's hard for it to get multiple recommendations when 1 recommendation would be enough. So, similar to the ratio idea for how many recommendations one can have, you could also have recommended ratio as a factor of number of finds and then set a threshold, such as (recommendations / finds * 10). eg 3 out of 10 finders recommend would give a recommended rating of 3, as would 1 recommendation out of 3 finders (roughly). Then one could set a threshold based on that ratio instead. Another option is to stick with the current system and highlight recommendations differently (bronze, silver, gold) based on how many recommendations there are. We could try variations of these in parallel and see what works best for people.

It should still be possible to browse an individual members recommendations, and it would be very cool to have the website identify cachers that have made recommendations similar to you. You could then use their indications as a guide.

We have the first idea already, just go to their cacher page:
http://geocaching.com.au/cacher/rediguana

We also have "Cachers who recommend this cache also recommend..."

Our todo list includes "What do my mates recommend" and "Cachers who recommend the caches I recommend also recommend...". Whew!

We need a shorter word for recommend. I "dig" this cache, I "praise" this cache. Is there a verb to go with "bonzer"? I "bonze" this cache.

- Rog

[Nemesis]

I like Mind Socket's idea of weighting the number of recommendations for a cache by the number of times that it has been found.

I also like the idea of using bronze, silver, and gold text to tag recommended caches.

I'm keen on a system that ensures a roughly constant proportion of caches gets each of bronze, silver, and gold tags within each state.

So, I propose that any cache that gets one or more recommendations gets at least bronze tag, this allows the recognition of new or very difficult caches that don't have many finds yet. To get a silver tag, a cache would need to be in the upper x% of recommended caches within a state, where the goodness of a caches recommendation is the number of recommendations divided by the number of finds (a percentage of 20 may work, provided that at least 20% of caches have been recommended ). A gold tag would be awarded in the same way as the silver, but with a higher threshold (maybe the top 5% of recommendations?). So, up to 80% may get a bronze tag, but I doubt it; about 15% would get silver; and 5% would get gold.

Edited so that final percentages add to 100%.

[rediguana]

For that reason, I'm less inclned to think it's an informed recommendation but it would be nice to reduce the number of these with some sort of ratio.

Agree, perhaps it would be something like (finds - starting limit) / x to determine the number of recommendations that can be made by a cacher. I would be expecting that a cacher should have to find somewhere between 20-50 cachers before they can start recommending caches. This number is set to x, so that only once you've found, in the case of x=20, 30 caches are you entitled to recommending a cache.

If the overall system is designed to produce the top 5 or 10% of caches, then we probably need to ensure that 10-20% of finds are able to be recommended, which would suggest a ratio of 1:5-1:10. This would ensure that there are enough recommendation being made that the best caches are likely to receive multiple recommendations to draw them apart from the others.

Again I agree for the same reason, but I'm not sure that it shouldn't be recognised at all. There is the issue of how often a cache is visited.

This is were we must look at how gold caches would be used. In the discussion we've had recently on the forums, the main use we've identified is when you're travelling to an area you don't normally cache, and want to identify the best caches to visit.

Lets face it, if you're hard core you're going to attempt to find all caches nearby, so recommendations nearby don't mean much. Its more about marking these caches so that when cachers come from out of your area, then locals have recommended the cream of the crop.

Therefore, the 4 day hike required caches will not be recommended in this manner, and will not achieve gold, unless enough people think it is deserving to attempt.

Whilst proportionality would be good, I think there needs to be a minimum set, such as three finds for any cache. A cache that only has one or two recommendations, no matter how remote or least visited does not warrant getting gold just because 1 person out of x attempts recommends it. By having a minimum of three, it removes some of the personal subjectivity involved in recommending a cache.

I would steer clear of recommendations as a ratio of finds as a rule because not everyone is going to recommend caches on geocaching.com.au because some will not even be aware of the site! In addition, those that object to recommending caches - and yes they exist - with drag the numbers down by deciding not to contribute. We have to cater for this. Which is why I think we should focus on absolute recommendations, not as proportions of found logs.

I thought about the B/S/G idea, but it might be just a little complex. I could go with Gold (say top 5%) and Silver (top 5-15%), but I think for simplicity it would really need to be kept down to just one - one saying this is a good cache and has been recommended by a number of people.

We also have "Cachers who recommend this cache also recommend..." Our todo list includes "What do my mates recommend" and "Cachers who recommend the caches I recommend also recommend...". Whew!

Looking forward to more!

We need a shorter word for recommend. I "dig" this cache, I "praise" this cache. Is there a verb to go with "bonzer"? I "bonze" this cache.

I rate this cache? Over here it would be choice. As in "That's a choice cache, eh?!"

Seriously, I hope that gc.com.au can improve what its got because it could become a kick-ass system.

Cheers Gav

[Team Red Devil]

I've clicked on a cache that appears recommended by a cacher with only 10 finds. For that reason, I'm less inclined to think it's an informed recommendation but it would be nice to reduce the number of these with some sort of ratio. Just as 10 is an arbitrary number now, picking an appropriate ratio will be just as arbitrarty. Maybe it could be done such that the average number of finds per cacher would allow for 10 recommendations.

<br>
<br>
Perhaps this could be done by assigning say a 'Top Three' for cachers with 10 cache finds, 'Top Five' for up to say....25 cache finds, and work in 5 choice increments as cache finds increase. So once you have a Top Five, and you have reached 26 cache finds, you'd have a top ten. Then you (read: Intellectual Programming Types) could allocate say, 5 'Top Caches' for every say....15 caches found. So each time you got an increment of 15 cache finds- your 'Top Cache' Allowance would increase by five choices. <br>
<br>

you could also have recommended ratio as a factor of number of finds and then set a threshold, such as (recommendations / finds * 10). eg 3 out of 10 finders recommend would give a recommended rating of 3, as would 1 recommendation out of 3 finders (roughly).

<br>
<br>
My suggestion: What about going basic and getting people to rate the caches ala 'Olympic' style. Bronze, Silver & Gold. You could then just have something that looks a little like the below placed on the cache page.<br>
<br>
This cache has been found by 62 cachers*. <br>
Ratings:<br>
Gold:24<br>
Silver:15<br>
Bronze: 10<br>
*Please note: this cache may not have been rated by ALL its finders. <br>
<br>
Does that make sense? So even if it is a remote cache- its rating will reflect its excellence as a cache- because it could say found by one cacher- and have its Gold Rating at 1.<br>
<br>
Hope that made sense. <br>
<br>
Marie<br>
*Love is a battlefield-Pat Benatar*

[rediguana]

I'm keen on a system that ensures a roughly constant proportion of caches gets each of bronze, silver, and gold tags within each state.

Yep, it has to work toward being the top x% in each state. And these are the thresholds to identify the most popular caches.

So, I propose that any cache that gets one or more recommendations gets at least bronze tag, this allows the recognition of new or very difficult caches that don't have many finds yet.

As you suggest later though, we'd end up swimming in Bronze tags and devaluing them. And given that some couples will be two logs in themselves, I suggest to reduce the subjectively somewhat, that a Bronze cache would be defined by having a minimum of three recommendations. I'd consider going to two, especially for states that are not as active in geocaching terms,

Where the goodness of a caches recommendation is the number of recommendations divided by the number of finds ...

See my previous post, because we can't rely on getting a certain return rate on recommendations, we shouldn't use this as a factor. Instead I think we need to focus on the number of recommendations achieved. Remember we have geocachers in New Zealand that are not aware of gc.com.au. There are probably similar in Australia that choose to use the secondary site. Those logs will show up, but they will never contribute a recommendation.

An extremely simple algorithm would be something like...

1. For each state, sort all caches by absolute number of recommendations DESC
2. Assign top 5% Gold
3. Assign next 10% Silver
4. Assign next 10% and/or 3+ Recs Bronze

But since we will potentially be lacking recommendations because some will choose not to make recommendations, we need to give out more to those that are prepared to make recommendations so that sorting by recommendations is more easily achieved.

Cheers Gav

[Nemesis]

...I would steer clear of recommendations as a ratio of finds as a rule because not everyone is going to recommend caches on geocaching.com.au because some will not even be aware of the site! In addition, those that object to recommending caches - and yes they exist - with drag the numbers down by deciding not to contribute...

With the system that I described above, it doesn't matter if only a limited proportion of cachers contribute, as the weighted rating threshold for silver and gold tags are calculated to select a certain proportion of caches. The more cachers that contributed, the higher the validity of the ratings, however.

I agree that we should get a number of recommendations that are proportional to our number of finds, for the system to work properly. Maybe a fixed percentage between 10 and 50% would work, with a threshold between 2 and 10 finds (i.e., the maximum number or recommendations would be the number of finds multiplied by the proportion of finds to use, and then truncated to the nearest whole number). A proportion as high as 50% would work because more than one cacher would recommend some caches, and some cachers would not use their ratings (so maybe only 60% or so of caches would get a bronze recommendation or higher). I wouldn't recommend allowing cachers to recommend the same number of caches as they have found, they need to be forced to choose some king of sensible criterion for a bronze rating (i.e., a 50% cut off?).

Edit to correct the intentional mistake in the expected bronze percentage.

[Team Red Devil]

As an addendum to my comment- in order to find the 'Top' caches in the state (for display on the gc.com.au pages) you could do it state by state so that a proper perspective is given in relation to the amount of caches/cachers. for example: <br>
<br>
Top Rated Caches<br>
NSW-<br>
Gold: 'Marie's Cache 01'- 210 (the number is reflective of its votes)<br>
Silver: 'Scott's Cache 04'- 162<br>
Bronze: 'Rhi's Cache 02' - 130<br>
<br>
You could do this for each state- maybe on a page of its own. Daily updates would be fine I think. I'm sure you get the drift....<br>
<br>
Personally, I would have trouble segregating people based on this percentage, times the square root of one hundred nine, divided by twenty nine, plus 2. (Sarcasm) And trying to work it out looks like a logistical nightmare. I think simple is best. And to counter people who are newbies just plotting Gold cache votes on all their found caches, their personal statistics would appear on that persons personal cache page- (So if we were new, and logged 16 caches, and 'Golded' all of them- it would show up when people clicked on our public 'statistics' page) then people could see for themselves that those votes were a bit odd, and decide for themselves on the overall total of votes whether or not to do the cache. At least in theory. <br>
<br>
Marie.

[rediguana]

My suggestion: What about going basic and getting people to rate the caches ala 'Olympic' style. Bronze, Silver & Gold. You could then just have something that looks a little like the below placed on the cache page.

I think that adds to the complexity. A yes/no recommendation is easier to make than gold/silver/bronze or no. I'm keen for this to be as easy a system to use and understand as possible (except for the server magic of course )

Cheers Gav

[Nemesis]

...An extremely simple algorithm would be something like...

1. For each state, sort all caches by absolute number of recommendations DESC
2. Assign top 5% Gold
3. Assign next 10% Silver
4. Assign next 10% and/or 3+ Recs Bronze

But since we will potentially be lacking recommendations because some will choose not to make recommendations, we need to give out more to those that are prepared to make recommendations so that sorting by recommendations is more easily achieved.

That's nice and simple to understand.

One problem with using the absolute number of recommendations for a cache in step one, is that the number of recommendations is confounded with the popularity of the cache (i.e., the more popular the cache, the more finds, and potentially the more recommendations it gets). You can't assume that the cache is more popular because it's better than other caches, it may be that it's simply easier to find and/or nearer to a centre of population and/or has been there longer. Consider rural caches and/or caches that are difficult to get to, they may get fewer finds and hence recommendations, even though they are just as good as corresponding urban caches.

On the other hand, if we consider using the ratio of the number of recommendations to the number of finds for a cache as the measure of goodness (in step one), we then avoid some of the unwanted effects of popularity. I must admit that there are some potential problems with this approach too, but they are not as serious. Consider the situation where only one person has found a cache, and then recommends it, the cache would get a rating of 1.0. Compare it to a cache that has 10 finders, but is recommended by only two of them (some cachers don't know about the gc.com.au recommendations system, or don't care), then the rating is only 0.2. The problem is one of increased quantisation noise with fewer observations. One way around the problem is to ignore any caches that have had, let us say, two or fewer visits (the threshold would have to be chosen so that the rating of a single cacher could not push the cache up to a silver or gold rating, it depends on the distribution of ratings). Maybe we could consider awarding caches that don't make that threshold, but are recommended, a bronze rating (just to acknowledge that the cache may have some merits)?

One advantage with replacing the minimum number of recommendations with a minimum number of finds in step four, is that single recommendations are not wasted, even where the number of finds is relatively small. If the number of finds is large, then a single recommendation gets diluted, and doesn't affect the final rating a whole lot (as it should be).

Edit to fix some of my grammar and terminology.

[Mind Socket]

recommendations squared divided by finds

1 recommendation, 1 find = 1
2 recommendations, 2 finds = 2
5 recommendations, 10 finds = 2.5
2 recommendations, 30 finds = 0.13
20 recommendations, 30 finds = 13

Pretty resilient to noise and low-find problems, but tends to cluster things a bit. I'll play with the data and some different (simple) measures sometime* and see what gives a good spread without low-find issues. We can apply the same ideas to any list whether it be recommended, watched, planned, ...

- Rog.

* where sometime between now() and infinity

[Nemesis]

Nice transformation Mind Socket. I was going to suggest something similar as a compromise, maybe by dividing the number of recommendations by the square root of the number of finds, that would favour more popular caches to some extent.

[Team Red Devil]

And if you divide that number proportionate to the amount of mcdonalds crap toys times 650 divided by 24 plus 20 equals the sum of 304 times 289 plus 65.23455 including the ratio of 234.2 to 1 which in turn equals how ridiculous all that formulation and counting is. <br>
<br>

A yes/no recommendation is easier to make than gold/silver/bronze or no. I'm keen for this to be as easy a system to use and understand as possible (except for the server magic of course )

<br>
<br>
Not exactly sure how a yes or no would fit in, and how it would measure anything about the cache. <br>
<br>
I am wondering if perhaps my method was just so simple that it went over the heads of everyone.Or maybe I'm just a moron. Probably the latter...oh well. I'm about to edit the other post I made which was 'statistic' based- and bought into the overly formulated debate- With the method I devised- it would be simple, it would provide the needed information and it wouldn't be that hard.<br>
<br>
For example (this would be on the 'log' page when you log your find):<br>
<br>
Would you rate this cache:<br>
Gold:*Insert little dot button here*<br>
Silver:*and here*<br>
Bronze:*and here*<br>
<br>
Easy Peasy. Then, if people go to your cache page- they could see a topic like *Cache Ratings* and this would include all the caches you'd done, and what you rated them- (not unlike the 'recommend' pages now). And if a cache was more popular than another for example NOT a four day trek through wilderness, but a quick pick up cache- it may get more votes, and would therefore only be rated highly, but I think the only people who'd really care are the total stats freaks. It would just be more of an indicator that that cache had been found more than the one that was a Gold cache that was a four day trek- not unlike the current 'Top Ten Caches' that appears on the front page of gc.com.au at the present moment. <br>
<br>
Perhaps the powers that be could create a search system where you could search for gold caches only in a search- then those who wanted could check out those gold caches for themselves. <br>
<br>
There would be NO need for long winded formulations and calculations- at least not so far as I can see. It would be simple. If I search for a cache in my area, and I see that three people have gone to it, and it only has ONE gold rating- then I know either the people are too lazy to fill it in, or don't give a shyte. But one person thought it awesome enough to list it as a Gold. <br>
<br>
Anyway- just my opinion. But hey- I'm just a newbie, and I know nothing about programming. If I can't make sense of the formula, nor get the gist of what you are all referring to- then like I said- I must be a moron- because with something like that- you need to know where and how the statistics are created in order to understand their relevance. At least thats been my experience.<br>
<br>
Marie

[swampgecko]

Sorry I have to disagree with all of the above... why over complicate a system that seems to be working?... I check two things when I look at the recommended caches, who recommends it and how many people are recommending it. Self recommenders have been weeded out, so that makes it easier.

If one cache one has one person recommending it why penalise it? that is what you are suggesting(IMHO). There maybe a good reason for why only one person has recommended it.. ie the aforementioned 4 day hike as such.

I sympathise with the issue of only 10 recommendations for each cacher to use, As I have a hard time with my list all the time, but the caches I have on my list are there because I believe they are the best examples around of the cacher's craft.. for whatever reason....

Please leave the system the way it is... it works for me.....