WCM Network: 
 


Go Back   mmorpgforum.net > Machinima > WarcraftMovies.com > Suggestions
Register Projects FAQ Members List Calendar Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Display Modes
Old 04-24-2008, 08:41 PM   #1
Abyzz
Initiate
 
Join Date: Dec 2006
Posts: 28
Abyzz is an unknown quantity at this point
Spicing up the new rating system

The overall issue with rating on WCM - one that you've gone to a great extend trying to counter with the new comment-on-comment system (may I call it CCS?) - is that the community seems to favor people with similar opinions rather than those who try to contribute something to the discussion. If someone goes against the current by constructively evaluating a movie that everyone seem to like (and vise versa), that comment will receive negative feedback, while overall useless comments stating nothing but the four-lettered "nice" receive good feedback. This promotes a simple and unsophisticated sort of response, and can not possibly be the aim.

A way to combat this could be to extend the CCS with rankings based on feedback on submitted comments. Those who consistently provide useful comments should stand out and inspire people to take their inputs more seriously, and while this would emphasize constructive feedback in a clearly positive way it would also encourage the community to take voting more seriously. After all, who wouldn't want to stand out and get that sweet colored text?

CCS is a great initiative, but I think it can become even better. Destinct the good 'raters' from the bad and inspire the community to think more carefully when voting - thorough feedback should come before superficial ones. Constructiveness over criticism.

Last edited by Abyzz : 04-24-2008 at 11:20 PM.
Abyzz is offline   Reply With Quote
Old 04-25-2008, 03:12 AM   #2
Dwârv
Senior Member
 
Dwârv's Avatar
 
Join Date: Sep 2006
Posts: 267
Dwârv is unknown at this point
Yeah, you touch a weak point here, Abyzz.
I also wonder why some comments are hidden - they include constructive crtitic and don't use foul language.It's just the simple fact that they oppose the mainstream that get's them "minus" clicks.Vice versa,as Abyzz said, there are the simple "well done,gieve moar plz" comments that get highlighted.
Hmmm.
Another way to straighten this out could be a limitation that not everyone is allowed to give a minus or plus.Somehow similar to the ability of rating at all, but with a higher number of movies needed to be watched/downloaded/ratet-whatever.

Another thing I find irritating that from time to time I find two or even three times the same rating in a row (example:Anarchyx on "Chronicles of the annoying quest" :three times 5 stars.Don't want to question the 5 stars,but the fact that there are three ratings by one person.This will make the rating unbalanced).
Is this working as intended?

Regards

Dwârv
__________________
RunningDwârv productions
Dwârv is offline   Reply With Quote
Old 04-25-2008, 03:46 AM   #3
lajonsky
Initiate
 
Join Date: Jan 2006
Posts: 1
lajonsky is unknown at this point
Pretty much exactly what I came to the forums to post about. Earlier today, I left a comment for Tosan's newest movie. I rated highly on all fronts except for A/V Quality, and outlined why ..

"Audio is not compressed, which is one of the reasons the video is so large. The other is that the video bitrate is too high for a resolution so small. To put it in perspective, Nihilum's Illidan kill video had about the same bitrate as your video, but it had a resolution of 1680x1052."

Because I didn't praise his movie like most within the 70 pages of comments, I got down rated to oblivion. My comment is probably one of the lowest rated on there right now, and it's starting to be that way with any others that make posts like it. You might as well get a warning page before posting that says "Criticism, especially the constructive type, is not welcome."
lajonsky is offline   Reply With Quote
Old 04-25-2008, 07:54 AM   #4
Abyzz
Initiate
 
Join Date: Dec 2006
Posts: 28
Abyzz is an unknown quantity at this point
And yet without constructive criticism, what is there to rate for? I hope (a fool's hope maybe) that the author actually reads my review and take at least some of it to heart for future creations - I will have hammered Alt+F4 long before remotely considering contributing with a thousand and seventh "O my GOD I want your BABIEZ!", as there are obviously others up for that task.

I find that the problem is twofold:

1) Ratings have inflated at a pace comparable to that in Zimbabwe in 2007, and an average 3 stars movie is now in fact a movie that should receive punishment for ever having stalked the earth. As a direct consequence of this, any attempt to rate fairly is looked upon as pessimistic and demeaning, and is with the new CCS blown to pieces by other raters.
2) The community seems to favor others of same opinion rather than those with a message: "there is not a fragment of logic in this bloke's arguments, but he gave the same number of stars so he'll get a +!". Going against the current (e.g. actually trying to make suggestions for improvement) will suffer because of CCS.

The ranking system could be improved in many ways, but here are a few ideas.

Author rating: The movie creator has special weight when commenting on the comments made in his/her movies. For example: authors altar the comment score by 2 or 3 points rather than the usual 1, so that his/her liking will immediately result in either hiding or highlighting comments. Of course there's a pitfall in this, since theoretically the author could be zealous about the product and therefore remove any sign of critique thus distorting the feedback. Still, such approaches would be discovered very soon and I believe it will regulate itself.
Rater promotion: Let's inspire more to rate, and to rate more seriously. The improved author-based CCS will result in an overall score for each commenter (which is the sum of negative and positive feedback), and when that score reaches a certain level, the member is promoted to "Enthusiast", with additional rights in commenting. The rights could be plentiful - it could be automatic comment highlighting upon posting, the option to make make breaks in the text field (following the thought that better raters are more thorough raters and use more text); it's a balance issue, but just make sure to make it worthwhile for the lot of us to rate seriously. As a derivative you, WCM, will get an easy overview on who's the 'bad guys'.
Hall of Fame: The 50 raters with highest CCS scores are added to the Hall of Fame in their own "Best Raters" category.
Award Icon: An icon on top of the movie where you see "2X Platinum" etc. is added if a certain amount of best raters have rated the movie positively. Basically if those who take rating most seriously agree to like a movie there's a good chance it's worth watching, so this should work as a promotion function for the movie.

Feel free to share your ideas and maybe we can narrow it down into something useful and that's also realistic for WCM...

Last edited by Abyzz : 04-25-2008 at 06:37 PM.
Abyzz is offline   Reply With Quote
Old 04-28-2008, 09:40 AM   #5
Uzbeki
Moderator
 
Uzbeki's Avatar
 
Join Date: Sep 2006
Posts: 1,261
Uzbeki is unknown at this point
Quote:
Originally Posted by Dwârv View Post
Another thing I find irritating that from time to time I find two or even three times the same rating in a row (example:Anarchyx on "Chronicles of the annoying quest" :three times 5 stars.Don't want to question the 5 stars,but the fact that there are three ratings by one person.This will make the rating unbalanced).
Is this working as intended?
I am taking a look at this now. During the last month it has happend to around 5 ratings. Not a huge problem but I believe it happens when our databases are slightly out of sync with each other. It has happened a few times the at the end of last week when we were upgrading some of the database servers. I will see what we can do do counteract this.
__________________
Founder and Lead Admin
WarcraftMovies.com
Uzbeki is offline   Reply With Quote
Old 04-28-2008, 10:06 AM   #6
Uzbeki
Moderator
 
Uzbeki's Avatar
 
Join Date: Sep 2006
Posts: 1,261
Uzbeki is unknown at this point
Quote:
Originally Posted by Abyzz View Post
Author rating: The movie creator has special weight when commenting on the comments made in his/her movies. For example: authors altar the comment score by 2 or 3 points rather than the usual 1, so that his/her liking will immediately result in either hiding or highlighting comments. Of course there's a pitfall in this, since theoretically the author could be zealous about the product and therefore remove any sign of critique thus distorting the feedback. Still, such approaches would be discovered very soon and I believe it will regulate itself.
Although this sounds like a good idea I have doubts about it working. A majority of the authors would use their extra points to hide low ratings. I open to it and we will see how the other changes (see further down) we are doing will impact the system.

Quote:
Originally Posted by Abyzz View Post
Rater promotion: Let's inspire more to rate, and to rate more seriously. The improved author-based CCS will result in an overall score for each commenter (which is the sum of negative and positive feedback), and when that score reaches a certain level, the member is promoted to "Enthusiast", with additional rights in commenting. The rights could be plentiful - it could be automatic comment highlighting upon posting, the option to make make breaks in the text field (following the thought that better raters are more thorough raters and use more text); it's a balance issue, but just make sure to make it worthwhile for the lot of us to rate seriously. As a derivative you, WCM, will get an easy overview on who's the 'bad guys'.
To some extent we are doing this already. All persons with 75 or higher fame gets their ratings automatically. One way of improving the power of those with higher fame (ex 50+) would be to give them a +/- comment score of 2 instead of 1.

Quote:
Originally Posted by Abyzz View Post
Hall of Fame: The 50 raters with highest CCS scores are added to the Hall of Fame in their own "Best Raters" category.
We could create a "Top 50" list for this yes.

Quote:
Originally Posted by Abyzz View Post
Award Icon: An icon on top of the movie where you see "2X Platinum" etc. is added if a certain amount of best raters have rated the movie positively. Basically if those who take rating most seriously agree to like a movie there's a good chance it's worth watching, so this should work as a promotion function for the movie.
I like the concept of this idea. The question is if it will differ much from the "Top Rated this Week" achievements we hand out every week. I will do some testing when get more data.

Today we are also going to change the +/- requirements for hiding and highlighting comments. From now on it will be -3 for hiding comments (instead of 2) and +5 for highlighting comments (instead of +3).

We are also thinking about hiding "irrelevant scores" of -2,-1,0. (positive ratings too?) I would also hate if someone "downrated" my constructive comment but it is irrelevant since the comment will not be hidden etc and it removing those irrelevant score will remove a lot of the annoyance of it.

Also keep in mind that even hidden ratings count until they have been removed and that all hidden comments are moderated by intelligent people who will not warn or remove legit criticism.
__________________
Founder and Lead Admin
WarcraftMovies.com
Uzbeki is offline   Reply With Quote
Old 04-28-2008, 11:33 AM   #7
Uzbeki
Moderator
 
Uzbeki's Avatar
 
Join Date: Sep 2006
Posts: 1,261
Uzbeki is unknown at this point
We now have a Top 50 list for voters in terms of points.
http://www.warcraftmovies.com/topten...rt=vote_points
__________________
Founder and Lead Admin
WarcraftMovies.com
Uzbeki is offline   Reply With Quote
Old 04-28-2008, 01:13 PM   #8
Dwârv
Senior Member
 
Dwârv's Avatar
 
Join Date: Sep 2006
Posts: 267
Dwârv is unknown at this point
I am in the list,waiter, bring some champagne!!

The "Award Icon" idea brought up by Abyzz looks good to me.
And changing the +/- requirements was definitly a step in the right direction.
__________________
RunningDwârv productions
Dwârv is offline   Reply With Quote
Old 04-28-2008, 03:39 PM   #9
Abyzz
Initiate
 
Join Date: Dec 2006
Posts: 28
Abyzz is an unknown quantity at this point
Quote:
Originally Posted by Uzbeki View Post
Although this sounds like a good idea I have doubts about it working. A majority of the authors would use their extra points to hide low ratings. I open to it and we will see how the other changes (see further down) we are doing will impact the system.
As said earlier this could indeed lead to your mentioned problem, and having to make separate systems for author voting seems a little too complicated. It boils down to this really: are ratings purely meant to indicate whether or not a movie is worth watching, or are they also minded at communicating with the author to improve the content for future releases? I guess it's a combination, but if we could somehow enforce constructiveness over criticism from an author perspective I reckon much is achieved. The main problem is that this is only a comment system, and the settings greatly limits our (the visitors) abilities to respond.

How about - this may be a wild shot (so if you run away screaming I forgive you!) - dynamic forum threads supporting the releases? Say I submit a movie called "Sunwell Sabotaged"; doing so will automatically create a thread of same name (with me, the creator) as author. This thread would be added to a new "Released Movies Discussion" forum, where those interested could get far more into detail about discussing that release. This would also have obvious benefits from a community involvement perspective, and would undoubtedly breath new life into other forum activities.

Quote:
To some extent we are doing this already. All persons with 75 or higher fame gets their ratings automatically. One way of improving the power of those with higher fame (ex 50+) would be to give them a +/- comment score of 2 instead of 1.
That's a good idea. I'm not sure how your fame system works towards weighing comments against other factors, but the 'best' voters should receive higher weight, so that they at a faster pace get influence to affect the community positively.

Quote:
We could create a "Top 50" list for this yes.
Thank you. It looks good, and I'm sure it will have a positive effect.

Quote:
I like the concept of this idea. The question is if it will differ much from the "Top Rated this Week" achievements we hand out every week. I will do some testing when get more data.
It was meant as a way to filter the noise that is the inflation on rating. IMDB.com which is probably one of the biggest rating communities in the world have this "Top 1000 Voters" concept, where their votes are separate in the statistics. The pattern, beyond statistical interference, shows that the top raters are also the toughest raters, rating lower than any other segment of raters in the community. I see no reason why this should not be the case here, and with this icon award the visitor could rest more assured that the listed rating does in fact reflect the content of the movie.

Quote:
Today we are also going to change the +/- requirements for hiding and highlighting comments. From now on it will be -3 for hiding comments (instead of 2) and +5 for highlighting comments (instead of +3).
Brilliant!

Quote:
We are also thinking about hiding "irrelevant scores" of -2,-1,0. (positive ratings too?) I would also hate if someone "downrated" my constructive comment but it is irrelevant since the comment will not be hidden etc and it removing those irrelevant score will remove a lot of the annoyance of it.
Question is, which system decides what's relevant and what's not?

Last edited by Abyzz : 04-28-2008 at 11:22 PM.
Abyzz is offline   Reply With Quote
Old 04-28-2008, 04:18 PM   #10
Abyzz
Initiate
 
Join Date: Dec 2006
Posts: 28
Abyzz is an unknown quantity at this point
Another thing that could really help make us raters' lives easier would be some sort of a tracking system on our comments. Upon commenting I see there's a sweet "Rating added!" tag next to the movie title, but when refreshing that tag is removed, and when pushed 22 pages back it's really quite annoying a week later to scroll through it all to find out whether or not I commented on this particular movie. The last 20 downloads displayed in the profile is insufficient in solving this problem.

The tag next to the title should remain for all eternity, and to make things even easier how about turning the tag into a link that would direct you to the comment (if there are several comments, linking to the last one with rating included). Maybe even further this maneuverability by formatting the text on releases on the front page based on whether or not you have rated (say the title gets an 'R' for rated at the front). If this goes towards too much information simply make it optional in the profile settings, so that it only displays for those having "Mark Rated Movies" checked.

Last edited by Abyzz : 04-28-2008 at 05:55 PM.
Abyzz is offline   Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Forum time (GMT) 09:48 PM


Powered by vBulletin® Version 3.6.12
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.