Spicing up the new rating system

[Abyzz]

The overall issue with rating on WCM - one that you've gone to a great extend trying to counter with the new comment-on-comment system (may I call it CCS?) - is that the community seems to favor people with similar opinions rather than those who try to contribute something to the discussion. If someone goes against the current by constructively evaluating a movie that everyone seem to like (and vise versa), that comment will receive negative feedback, while overall useless comments stating nothing but the four-lettered "nice" receive good feedback. This promotes a simple and unsophisticated sort of response, and can not possibly be the aim.

A way to combat this could be to extend the CCS with rankings based on feedback on submitted comments. Those who consistently provide useful comments should stand out and inspire people to take their inputs more seriously, and while this would emphasize constructive feedback in a clearly positive way it would also encourage the community to take voting more seriously. After all, who wouldn't want to stand out and get that sweet colored text?

CCS is a great initiative, but I think it can become even better. Destinct the good 'raters' from the bad and inspire the community to think more carefully when voting - thorough feedback should come before superficial ones. Constructiveness over criticism.

[Dwârv]

Yeah, you touch a weak point here, Abyzz.
I also wonder why some comments are hidden - they include constructive crtitic and don't use foul language.It's just the simple fact that they oppose the mainstream that get's them "minus" clicks.Vice versa,as Abyzz said, there are the simple "well done,gieve moar plz" comments that get highlighted.
Hmmm.
Another way to straighten this out could be a limitation that not everyone is allowed to give a minus or plus.Somehow similar to the ability of rating at all, but with a higher number of movies needed to be watched/downloaded/ratet-whatever.

Another thing I find irritating that from time to time I find two or even three times the same rating in a row (example:Anarchyx on "Chronicles of the annoying quest" :three times 5 stars.Don't want to question the 5 stars,but the fact that there are three ratings by one person.This will make the rating unbalanced).
Is this working as intended?

Regards

Dwârv

[lajonsky]

Pretty much exactly what I came to the forums to post about. Earlier today, I left a comment for Tosan's newest movie. I rated highly on all fronts except for A/V Quality, and outlined why ..

"Audio is not compressed, which is one of the reasons the video is so large. The other is that the video bitrate is too high for a resolution so small. To put it in perspective, Nihilum's Illidan kill video had about the same bitrate as your video, but it had a resolution of 1680x1052."

Because I didn't praise his movie like most within the 70 pages of comments, I got down rated to oblivion. My comment is probably one of the lowest rated on there right now, and it's starting to be that way with any others that make posts like it. You might as well get a warning page before posting that says "Criticism, especially the constructive type, is not welcome."

[Abyzz]

And yet without constructive criticism, what is there to rate for? I hope (a fool's hope maybe) that the author actually reads my review and take at least some of it to heart for future creations - I will have hammered Alt+F4 long before remotely considering contributing with a thousand and seventh "O my GOD I want your BABIEZ!", as there are obviously others up for that task.

I find that the problem is twofold:

1) Ratings have inflated at a pace comparable to that in Zimbabwe in 2007, and an average 3 stars movie is now in fact a movie that should receive punishment for ever having stalked the earth. As a direct consequence of this, any attempt to rate fairly is looked upon as pessimistic and demeaning, and is with the new CCS blown to pieces by other raters.
2) The community seems to favor others of same opinion rather than those with a message: "there is not a fragment of logic in this bloke's arguments, but he gave the same number of stars so he'll get a +!". Going against the current (e.g. actually trying to make suggestions for improvement) will suffer because of CCS.

The ranking system could be improved in many ways, but here are a few ideas.

Author rating: The movie creator has special weight when commenting on the comments made in his/her movies. For example: authors altar the comment score by 2 or 3 points rather than the usual 1, so that his/her liking will immediately result in either hiding or highlighting comments. Of course there's a pitfall in this, since theoretically the author could be zealous about the product and therefore remove any sign of critique thus distorting the feedback. Still, such approaches would be discovered very soon and I believe it will regulate itself.
Rater promotion: Let's inspire more to rate, and to rate more seriously. The improved author-based CCS will result in an overall score for each commenter (which is the sum of negative and positive feedback), and when that score reaches a certain level, the member is promoted to "Enthusiast", with additional rights in commenting. The rights could be plentiful - it could be automatic comment highlighting upon posting, the option to make make breaks in the text field (following the thought that better raters are more thorough raters and use more text); it's a balance issue, but just make sure to make it worthwhile for the lot of us to rate seriously. As a derivative you, WCM, will get an easy overview on who's the 'bad guys'.
Hall of Fame: The 50 raters with highest CCS scores are added to the Hall of Fame in their own "Best Raters" category.
Award Icon: An icon on top of the movie where you see "2X Platinum" etc. is added if a certain amount of best raters have rated the movie positively. Basically if those who take rating most seriously agree to like a movie there's a good chance it's worth watching, so this should work as a promotion function for the movie.

Feel free to share your ideas and maybe we can narrow it down into something useful and that's also realistic for WCM...

[Uzbeki]

Another thing I find irritating that from time to time I find two or even three times the same rating in a row (example:Anarchyx on "Chronicles of the annoying quest" :three times 5 stars.Don't want to question the 5 stars,but the fact that there are three ratings by one person.This will make the rating unbalanced).
Is this working as intended?


I am taking a look at this now. During the last month it has happend to around 5 ratings. Not a huge problem but I believe it happens when our databases are slightly out of sync with each other. It has happened a few times the at the end of last week when we were upgrading some of the database servers. I will see what we can do do counteract this.

[Uzbeki]

Author rating: The movie creator has special weight when commenting on the comments made in his/her movies. For example: authors altar the comment score by 2 or 3 points rather than the usual 1, so that his/her liking will immediately result in either hiding or highlighting comments. Of course there's a pitfall in this, since theoretically the author could be zealous about the product and therefore remove any sign of critique thus distorting the feedback. Still, such approaches would be discovered very soon and I believe it will regulate itself.

Although this sounds like a good idea I have doubts about it working. A majority of the authors would use their extra points to hide low ratings. I open to it and we will see how the other changes (see further down) we are doing will impact the system.


Rater promotion: Let's inspire more to rate, and to rate more seriously. The improved author-based CCS will result in an overall score for each commenter (which is the sum of negative and positive feedback), and when that score reaches a certain level, the member is promoted to "Enthusiast", with additional rights in commenting. The rights could be plentiful - it could be automatic comment highlighting upon posting, the option to make make breaks in the text field (following the thought that better raters are more thorough raters and use more text); it's a balance issue, but just make sure to make it worthwhile for the lot of us to rate seriously. As a derivative you, WCM, will get an easy overview on who's the 'bad guys'.

To some extent we are doing this already. All persons with 75 or higher fame gets their ratings automatically. One way of improving the power of those with higher fame (ex 50+) would be to give them a +/- comment score of 2 instead of 1.


Hall of Fame: The 50 raters with highest CCS scores are added to the Hall of Fame in their own "Best Raters" category.

We could create a "Top 50" list for this yes.


Award Icon: An icon on top of the movie where you see "2X Platinum" etc. is added if a certain amount of best raters have rated the movie positively. Basically if those who take rating most seriously agree to like a movie there's a good chance it's worth watching, so this should work as a promotion function for the movie.

I like the concept of this idea. The question is if it will differ much from the "Top Rated this Week" achievements we hand out every week. I will do some testing when get more data.

Today we are also going to change the +/- requirements for hiding and highlighting comments. From now on it will be -3 for hiding comments (instead of 2) and +5 for highlighting comments (instead of +3).

We are also thinking about hiding "irrelevant scores" of -2,-1,0. (positive ratings too?) I would also hate if someone "downrated" my constructive comment but it is irrelevant since the comment will not be hidden etc and it removing those irrelevant score will remove a lot of the annoyance of it.

Also keep in mind that even hidden ratings count until they have been removed and that all hidden comments are moderated by intelligent people who will not warn or remove legit criticism.

[Uzbeki]

We now have a Top 50 list for voters in terms of points.
http://www.warcraftmovies.com/topten.php?sort=vote_points

[Dwârv]

I am in the list,waiter, bring some champagne!!

The "Award Icon" idea brought up by Abyzz looks good to me.
And changing the +/- requirements was definitly a step in the right direction.

[Abyzz]

Although this sounds like a good idea I have doubts about it working. A majority of the authors would use their extra points to hide low ratings. I open to it and we will see how the other changes (see further down) we are doing will impact the system.As said earlier this could indeed lead to your mentioned problem, and having to make separate systems for author voting seems a little too complicated. It boils down to this really: are ratings purely meant to indicate whether or not a movie is worth watching, or are they also minded at communicating with the author to improve the content for future releases? I guess it's a combination, but if we could somehow enforce constructiveness over criticism from an author perspective I reckon much is achieved. The main problem is that this is only a comment system, and the settings greatly limits our (the visitors) abilities to respond.

How about - this may be a wild shot (so if you run away screaming I forgive you!) - dynamic forum threads supporting the releases? Say I submit a movie called "Sunwell Sabotaged"; doing so will automatically create a thread of same name (with me, the creator) as author. This thread would be added to a new "Released Movies Discussion" forum, where those interested could get far more into detail about discussing that release. This would also have obvious benefits from a community involvement perspective, and would undoubtedly breath new life into other forum activities.

To some extent we are doing this already. All persons with 75 or higher fame gets their ratings automatically. One way of improving the power of those with higher fame (ex 50+) would be to give them a +/- comment score of 2 instead of 1.That's a good idea. I'm not sure how your fame system works towards weighing comments against other factors, but the 'best' voters should receive higher weight, so that they at a faster pace get influence to affect the community positively.

We could create a "Top 50" list for this yes.Thank you. It looks good, and I'm sure it will have a positive effect.

I like the concept of this idea. The question is if it will differ much from the "Top Rated this Week" achievements we hand out every week. I will do some testing when get more data.It was meant as a way to filter the noise that is the inflation on rating. IMDB.com which is probably one of the biggest rating communities in the world have this "Top 1000 Voters" concept, where their votes are separate in the statistics. The pattern, beyond statistical interference, shows that the top raters are also the toughest raters, rating lower than any other segment of raters in the community. I see no reason why this should not be the case here, and with this icon award the visitor could rest more assured that the listed rating does in fact reflect the content of the movie.

Today we are also going to change the +/- requirements for hiding and highlighting comments. From now on it will be -3 for hiding comments (instead of 2) and +5 for highlighting comments (instead of +3). Brilliant!

We are also thinking about hiding "irrelevant scores" of -2,-1,0. (positive ratings too?) I would also hate if someone "downrated" my constructive comment but it is irrelevant since the comment will not be hidden etc and it removing those irrelevant score will remove a lot of the annoyance of it. Question is, which system decides what's relevant and what's not?

[Abyzz]

Another thing that could really help make us raters' lives easier would be some sort of a tracking system on our comments. Upon commenting I see there's a sweet "Rating added!" tag next to the movie title, but when refreshing that tag is removed, and when pushed 22 pages back it's really quite annoying a week later to scroll through it all to find out whether or not I commented on this particular movie. The last 20 downloads displayed in the profile is insufficient in solving this problem.

The tag next to the title should remain for all eternity, and to make things even easier how about turning the tag into a link that would direct you to the comment (if there are several comments, linking to the last one with rating included). Maybe even further this maneuverability by formatting the text on releases on the front page based on whether or not you have rated (say the title gets an 'R' for rated at the front). If this goes towards too much information simply make it optional in the profile settings, so that it only displays for those having "Mark Rated Movies" checked.

[Abyzz]

Looking for a response. Is it out there? Can it be found? Does it exist? *puts on eagle eye vision*

[Dwârv]

...but all he could see was the echoe of his desperate questioning,reflecting from the concrete walls of this devasted place....

[Abyzz]

..."place place place"...

[Uzbeki]

That's a good idea. I'm not sure how your fame system works towards weighing comments against other factors, but the 'best' voters should receive higher weight, so that they at a faster pace get influence to affect the community positively.


It was meant as a way to filter the noise that is the inflation on rating. IMDB.com which is probably one of the biggest rating communities in the world have this "Top 1000 Voters" concept, where their votes are separate in the statistics. The pattern, beyond statistical interference, shows that the top raters are also the toughest raters, rating lower than any other segment of raters in the community. I see no reason why this should not be the case here, and with this icon award the visitor could rest more assured that the listed rating does in fact reflect the content of the movie.


I am just writing a quick response here to let you know I have not forgotten about this, we have just been busy with a few other issues.

All voters have a rating weight assigned to them already. :) Newbie voters have little impact on the score.
For example http://www.warcraftmovies.com/pv.php?l=Dw%E2rv has 100 (max)
while http://www.warcraftmovies.com/pv.php?l=Abyzz has 61.

In short we have a lot of systems that makes sure that random voters impact on the outcome of the score is minimized, even though there are a lot of "spam" it does not count as much in the end.

I like the suggestion about having the top 100 (or something) voters have their own "award" each week. I will crunch the numbers and see how much it differs from the "general" opinion and present it here next week.

I will respond to the other suggestions after the weekend.

[Abyzz]

Thank you for clearing that one out for me Uzbeki, and thanks in general for being so busy all the time trying to improve the site for everyone's pleasure. It can't be said too often, but what you've created with WCM really is a remarkable achievement, and these days I reckon few don't know of its existence. How can we not honor that you're busy contributing to that!?

...So if me or some of the other forum monkeys at times seem a little impatient just think of us as grumpy old farts who haven't got around to train the 'Patience' skill for a wee too long - deep down we still love you and all your efforts towards making this a better place to be - though sometimes it's hard to admit! :D

[K21Nova]

I posted about this in another thread some time ago, but since this thread is continuing the discussion about the rating system, I thought I'd bring up my suggestion once again.

Now, this isn't a finalized system of rating that I'm suggesting, but rather just an idea about what the rating system could move towards.

I suggested a mutually exclusive system of ratings.

One system would be for the machinima-based ratings, where people who actually care about the video as a machinima would rate it for quality, editing, and these kinds of values.

The other system would give a few rating choices for the player's PvP prowess and such.

Now, I am well aware that at present WCM indeed has a skill point system alongside the machinima rating system, BUT...

What I was suggesting would make the rating systems mutually exclusive.

What this would mean would be that people would have to choose which way to vote...whether to give the author ratings for PvP skill etc, or whether be shown to the community as a machinima critic.

This way you couldn't give a video 5/5/5/5 (in machinima scores) and a skill point just because it's Niar. Or some similar situation.

Also, taking this idea further, we could have two "tabs" of comments. Perhaps give the "e-peen" community the topmost tab, the one with the PvP and skill related scores and comments, and then underneath that the actual ratings for the video itself, with it's own comment area.

Whether that's possible to do without going to extremes codewise, I don't know. My web knowledge stops at XHTML and CSS2.

The idea still remains though. Reading through the comments of the videos I download, which are quite often PvP videos, although I'm more of a storyline buff, I see so so SO many 5's and 1's with completely asinine comments which have absolutely nothing to do with the what the rating system is all about.

One reason why I believe people DO downrate and uprate is either to feel better about themselves and put down the other person in any way possible, OR they're acting a bit like fanbois. Not the ONLY reasons for downrating/uprating but I'm sure one of the biggest one.

Maybe with 2 separate and mutually exclusive rating systems these upraters and downraters would stick to the PvP side, and thus the value of the machinima scores would be a bit higher.

After all you can never get rid of all the people who take part in this downrating/uprating of videos, and even with a dedicated LEGION of people removing most of the 1's and 5's, there'd STILL be quite a few ratings which were given for nonsensical reasons having nothing to do with content and quality etc.

Anyway, that's an idea, and maybe it could spark something even better. :)

- Nova

[Abyzz]

Definitely an interesting idea you got there Nova, and I like your take on rating psychology. If you don't mind I'll try to give my view on it.

The issue about making exceptions and rating criteria for certain movies (one that has been discussed briefly in this topic (http://www.mmorpgforum.net/showthread.php?t=4399)), is that fundamentally by narrowing something down, you remove some of the receiver's freedom of choices. Making machinima ratings for machinima movies only would force me to rate PvP movies solely based on player prowess, and neglect the aspects of presentation that are still vital to any movie, regardless of genre. Giving a PvP movie top grades based on the speed of reflexes is like basing the purchase of a Ferrari on the amount of horsepower it has - there's more to it than that.

Then comes the issue with cross-genre movies. When you think about it not all movies are pure PvP or pure machinima, but have elements from other genres in them - elements that you as a rater need to be able to evaluate. And for that to work, you need more exceptions, more rating criteria, and ultimately a bigger headache in giving your verdict. It may be more representative for those who care to take it more seriously, but unfortunately this specie amongst WCM users is a rare one, and the efforts made by these individuals easily drown in the swarm of win/fail grades.

How could we counter these two aspects in any other way than to simplify things?

[K21Nova]

It's true that making people choose whether they rate "skill" or the video itself takes away their freedom to do both.

Then again, if you have both systems side by side, like you have now, you'll end up with a lot of 5/5/5/5 -ratings from people who think the guy in the video is skilled, even if the video itself is raw fraps footage with an ugly UI, made into a very basic video with Windows Movie Maker by slapping on some Disturbed, and thus would deserve most likely more of a 3 than a 5.

You know what I mean?

Perhaps if one's fame reaches a certain point they can rate both sides? Perhaps "good authors" and premium members can do so, etc?

Or perhaps some other system or idea that's not been mentioned yet. :)

I just wish that at least most of those "ZOMG ITS NIAR LOLZ I GIEF 5!11" and similar people, people who rate for the person in the video, for the skill or whatever, for a song or music in general, and mostly just ignore what the rates should be for, could be gotten rid of, or better yet, educated and conformed into actually serving the rating system instead of screwing with it.

I'll keep my thumbs up. Awesome site, nevertheless. :)

[Abyzz]

I think the problem is that you're trying to adjust the system to the flaws of human beings, and by doing so only increase the amount of limitations. If Niar gets an auto-5 then it's because he's done something in the past that in the viewers eyes makes him worth more than he is - it's a common feat really, to favor something you know is good. If you were handed a Coke, weren't you more prone to like the content than if you were handed, say, a Jolly? No system can change these psychological aspects.

The idea of using certain rankings to alter the possibilities to rate could work, and would help filter the useless spam, and make it less influential. Fundamentally, those who are more serious about the system should have more impact on it. There's already weighting in the impact of your rating though (depending on much more than just Fame), and to take it a step further is overkill I think.

I personally suggested that the "Skilled?" button were added to the rating perimeters, but that still doesn't solve the issue of not taking any (or every) perimeters seriously when rating, and to solve that you need to merge them. Put simply, forget all about these different variables and make one bar with, say, 10 stars in it that says "Rating". We might not be able to tell the audience that it's a good movie but with bad editing, but who cares anyway? How many won't download a movie that's rated 4.5 overall, if editing is only 3? Besides, we have the comments to elaborate should there be something worth noticing. Forget about complicating things, about giving special people advantages, adding perimeters for special movie genres, or trying to counter people's motives. There is no such system as a 100% accurate system, and embracing this fact I think it's best just to let the numbers do the talking. The more who rate, the more representative it'll be, and with an easier system chances are more will rate - and that their ratings will actually better match their impressions.

[Uzbeki]

Regarding the "Top Selected" highest rating during last week.
Minimum amount of voters: 5

Voters with over 50 fame
------------------------
title average amount
Tactical Guide - Illidan by Tamzin 5
Hunter Vs World 8 - Solo-ology 4.98
Thrall's Crib 4.95
The line must be drawn here! 4.89
First of May 4.86

Voters with over 65 fame
------------------------
title average amount
Hunter Vs World 8 - Solo-ology 4.96
The line must be drawn here! 4.93
Through the Roof 'n' Underground 4.84
Thrall's Crib 4.8
Nutty v1 - Mage PvP 4.78

Voters with over 75 fame
------------------------
title average amount
The line must be drawn here! 4.97
Through the Roof 'n' Underground 4.82
Nutty v1 - Mage PvP 4.8
BeoCraft Comparison 4.75
Untouchable Rogue 1: Mother Shahraz 4.6

Voters with over 85 fame
------------------------
title average amount
The line must be drawn here! 5


I looks like Fame ~75 might be a good number. Any suggestions to what we should call this new Achievement?

[Gedan]

There are several big sites that have adopted this part about easy rating, we are not in a stage right now where we are willing to remove our different variables but I totally agree with you Abyzz about making it too complex makes us lose votes..

[Uzbeki]

I think the problem is that you're trying to adjust the system to the flaws of human beings, and by doing so only increase the amount of limitations. If Niar gets an auto-5 then it's because he's done something in the past that in the viewers eyes makes him worth more than he is - it's a common feat really, to favor something you know is good. If you were handed a Coke, weren't you more prone to like the content than if you were handed, say, a Jolly? No system can change these psychological aspects.

The idea of using certain rankings to alter the possibilities to rate could work, and would help filter the useless spam, and make it less influential. Fundamentally, those who are more serious about the system should have more impact on it. There's already weighting in the impact of your rating though (depending on much more than just Fame), and to take it a step further is overkill I think.

I personally suggested that the "Skilled?" button were added to the rating perimeters, but that still doesn't solve the issue of not taking any (or every) perimeters seriously when rating, and to solve that you need to merge them. Put simply, forget all about these different variables and make one bar with, say, 10 stars in it that says "Rating". We might not be able to tell the audience that it's a good movie but with bad editing, but who cares anyway? How many won't download a movie that's rated 4.5 overall, if editing is only 3? Besides, we have the comments to elaborate should there be something worth noticing. Forget about complicating things, about giving special people advantages, adding perimeters for special movie genres, or trying to counter people's motives. There is no such system as a 100% accurate system, and embracing this fact I think it's best just to let the numbers do the talking. The more who rate, the more representative it'll be, and with an easier system chances are more will rate - and that their ratings will actually better match their impressions.

I think you make a few very good points here. We currently require at least 2 fame for people to vote. We are thinking about lowering it to 1 fame or removing it completely.

We are also looking at changing the rating categories of 4 with 1-5 stars to 1-2 sliders with 1-5 with 0.1 increments. One slider would be Overall rating and the other would be genre specific, such as "Skill" on PvP movies.

[Dwârv]

...We are also looking at changing the rating categories of 4 with 1-5 stars to 1-2 sliders with 1-5 with 0.1 increments. One slider would be Overall rating and the other would be genre specific, such as "Skill" on PvP movies.

All right, now we are talking - go for it..!

[Abyzz]

What scare me about those statistics is that they show an overall tendency to max out grades. I only have my own view on this, but to me a full 5 star movie is virtually a perfect movie, and those should be few and far between. What I'm seeing is (at least) 5 movies per week above 4.86 rating by those users who should have the best starting point to properly judge a movie. Be that as it may, in regards to a title I could be brash and suggest calling it "Famous" (e.g. "#2 Most Famous Week 22"), as Fame is basically what triggers the achievement.

I'll come back with a more thorough input once I'm done golfing. ;)

[Uzbeki]

What scare me about those statistics is that they show an overall tendency to max out grades. I only have my own view on this, but to me a full 5 star movie is virtually a perfect movie, and those should be few and far between. What I'm seeing is (at least) 5 movies per week above 4.86 rating by those users who should have the best starting point to properly judge a movie. Be that as it may, in regards to a title I could be brash and suggest calling it "Famous" (e.g. "#2 Most Famous Week 22"), as Fame is basically what triggers the achievement.

I'll come back with a more thorough input once I'm done golfing. ;)

Well we are going to change the rating system with sliders so you dont feel like you have to vote 4 or 5. You can vote 4.4 etc :)

But to think we can change a generation of youtube voters I feel is aiming a bit too high. We will never get a perfect rating system, there are just too many feelings involved in a gaming community like WoWs and a large portion of the voters are either fanboys or haters.

[Abyzz]

That may very well be, but I think it largely has something to do with ignorance about the given rating perimeters. Many people simply don't know what they should base their evaluation on when looking at editing, creativity etc., and therefore these ratings are merely displaying the general impression of the movie ("if I like it, editing must be 5!"). To think that people can somehow learn this is utopia, but I do believe much of it has to do with rating complexity, and that's an aspect you control.

Adding to that I'm not sure what you wish to accomplish with the new sliders. We can only be so precise, and I don't know about you, but I would never have a clue whether a movie should receive 4.1 in content or 4.2, and I wouldn't be able to justify either of them over the other. I mean think about it, that's basically 50 degrees for each four perimeters - even professionals (look at school systems) never have more than 10 degrees in which to rate. Or maybe I'm misunderstanding this slider concept?

[Abyzz]

I think you make a few very good points here. We currently require at least 2 fame for people to vote. We are thinking about lowering it to 1 fame or removing it completely.

We are also looking at changing the rating categories of 4 with 1-5 stars to 1-2 sliders with 1-5 with 0.1 increments. One slider would be Overall rating and the other would be genre specific, such as "Skill" on PvP movies.Removing fame-dependent permission to rate would be best I think. I doubt the ones with 1 Fame represent a large segment, so it shouldn't have any substantial impact - and that's where Rating Weight come in.

How would this genre specific rating work? I mean, there are lots of genres, and there are lots of movies that go across some of these. Is it not risky to let it all out on one figure (e.g. skill), when a good PvP movie is made up of so much more? Skill most definitely has an impact on the level of entertainment, but to me humor actually has an even bigger impact. If I find myself laughing throughout a movie, it can be poorly edited or display zero skills all I care, I would never hesitate to recommend it to others. Similarly, many other things are just as important as skills.

[Uzbeki]

We are still planning out the new rating system. We have been a bit busy with the release of Age of Conan but this is still def. on the roadmap.

[K21Nova]

Here's another good example why the current rating system fails miserably... I recently rated a video called "Mission 4.5 - Elemental Mage PvP" (http://www.warcraftmovies.com/movieview.php?id=73913) with a rating of 3/2/3/3. A low rating for a high-rated video, but I stick by my rating; the video itself deserved those numbers. The player IN it was skilled, and I never said otherwise. Here's my rating comment:

Content average; another "slap mp3 on fraps footage" video. Creativity below average; not even a single thing beyond fraps footage with mp3s. Editing average, quality average.

I don't think I was at all insulting the author with that rating, but I do think I was being honest.

Today I logged on, and noticed someone had sent me a private message. Nothing too bad, just saying it was worth watching etc. Well, I went to check my rating of the video.

My comment was at "-9" points and hidden.

I was not downrating because of the author.
I was not downrating because of music.
I was not downrating because of the class.
I was not downrating because of the race.
I was not downrating, period.

I gave an honest opinion of the video itself, but SKILL WORSHIPING FANBOIS and the like decided to vote my rating and comment down to oblivion.

Should I rate it 5, because it already has a rating of 4.50+ or because the player in it is skilled, even though the video is average, period? I personally don't think so. But apparently giving an honest opinion, if it's going against the fanboi masses, is useless, and by the popular fanboi vote your honest rating gets "disappeared".

This is why the rating system fails.

Unless of course you want it to be like this; quality of the video is irrelevant as long as the player in it is skilled and the video becomes popular?

[K21Nova]

As far as fame being an indicator of good ratings...

2 comments below mine there's a comment made by a person called "Wombat" (Wombat @ 08-05-25 03:54), giving full 5 points, and his comment says "Highskill!".

He has a fame of 60.

'nuff said?

[Abyzz]

I understand your concern, but as with this thread it is all about being constructive. Rating states the evaluation, and the comments explains that evaluation - hopefully - in a constructive manner. Saying that editing is average is a repetition of the score and doesn't really offer much in the sense of suggestions to improve, or a statement of why it is average. That said I don't believe your comment should receive anywhere near -9, but there seems to be a pack-instinct with the community in that if a comment is highlighted or blown to pieces it will emphasize itself until the point of absurdity.

And then of course there's the ones that completely abuse the system to prolong their figurative e-peens, awarding themselves 17 points in one comment stating "Cool." with the standardized 5/5/5/5 so that they top the Voters Points list.

But to return to the subject: if you feel the current system is unfair and unbalanced, what would you suggest to improve it? Anyone can spot a problem, but to come with a working solution is what really matters. :)

[Uzbeki]

One of the reason why its easy to spot flaws in the rating system is because we offer a lot of transparency. On for example YT you cant see what a person rates. You can only read the comment. Who knows what the person rated really? In fact most video sites does this.

Maybe we should just make all ratings 100% anonymous like everyone else does to make it impossible to criticize it? :P

[Dwârv]

...and delete the plus/minus button on comments as well? Cause this is what you would have to do to get the result you want: avoid criticizing.
With all it's weak spots I still think that the maximum transparency way is the one you should stick with.
@ Abyzz:...and yeah, this problem you mentioned about dopning your own comments (and top 50 voters position) made me already write a PM - unanswered by now.Like this the list makes no sense at all, but I think it's a problem WcM could fix (and I thought they allready did, until I was proofed wrong) - and in my humble opinion they should...or delete this list.

[K21Nova]

The thing about giving everyone the ability to influence the votes of others (with the - and + system), is that it's just a way to shut down minorities.

I mean, being in a minority is difficult as it is, but if you give the majority the power to remove your voice (by voting your comment/rating down to oblivion), then what's the point of even rating anything?

There are always minorities. Either the minority dislikes a movie, or likes a movie, but that's the whole idea behind a voting system; everyone votes, and in the end you get a result.

This sort of system kind of removes minorities and makes the system more and more black and white.

Add the fact that there indeed are...zounds of people giving full fives for no good reason, or for the complete wrong reasons, and if you on the other hand rate for the right reasons, and they don't like it, say goodbye to your rating.

Well.. I know I should give an example of a better system, but right now all I can think of is that in extreme cases the + - system doesn't work.

In extreme cases Mr X publishes a video, Mr X's followers and fanbase and whoever liked Mr X's moves give full fives, and along comes someone who thinks that fraps and mp3's and windows movie maker a good movie maketh not and this someone's voice and rating ends up voted down and disappearing.

What then would be a better system? Sigh, if only I knew. A more in-depth extensive rating questionnaire perhaps?

Ok, what about this?

Instead of having four different ratings, make it so that you can rate a video either 0, 1, 2, 3, 4 or 5, and if you REALLY care enough, you can go into a more extensive rating panel with like I dunno TEN different ratings.

This way most of the e-peen point givers just throw the 5 mainly because that's the easy way, and people who care will open up the actual rating panel and think about the ratings and rate accordingly.

Then perhaps show more information on the rating of the video, and somehow separate the "fast ratings" from the in-depth ones.

I dunno.

Something to remove the "ZOMG I GIEAF 5/5/5/5/5/5 CUZ ITZ MING AN I WUNA B LIEK MING WEN I GREW UP!111leventy" -votes :) (sigh, im tired...sorry)

[Dwârv]

K21Nova:...don't get it wrong - the rating will not disappear/be excluded, even if the comment get's flooded with minus clicks.
I personally open ALL hidden comments because I want to know for what reason the user got the minus clicks.So for me these comments/ratings don't disappear into oblivion, they are just one click away.

And lets face it:the more popular WcM becomes,the more childish fanboys will drop in,the more stupid comments and senseless ratings we will see here.This goes along with a massive load of frapsed PvP "movies",wich nobody but the maker him/herself cares about.
Pretty soon this site will be youtubed, like it or not.
Right now you can literally observe average quality of movies / ratings / comments / ratings on comments dropping down more every day.
I will stick with the dying corpse as long as I can bear it.

What I want to say is:WcM will not get rid of this, it will get worse and I can think of no rating system to get around this.

[K21Nova]

I just thought I'd draw your attention to the recent movie Solo Onyxia (http://www.warcraftmovies.com/movieview.php?id=76343).

I just went through about 20 pages of the comments there, and just about all I see is people giving it 1 star for reasons having nothing to do with the video itself, and calling the author names.

Also, almost any and all ratings of 3 and above have been voted down, some even down to -10 or less, even though the ratings themselves seem completely legit and most have very decent comments in them as well.

Just to give you some examples:

Nicely done. And clever to use a dead hunter to circumvent Blizzards inability to fix the Warder bug.

^ 3.8 rating, voted down to -9.

I kill onyxia all the time for gold. Normaly whit a group of 3 so its nice and easy, but the i did was 2 man, resto shaman(me) whit a retpaly. I realy enjoy that kind of movie and now i cant stop tinking of a way i could do a solo as resto shamy :)

^ 4.2 rating, voted down to -10.

Content3 Creativity2 Editing4 Quality5 Gratz for the kill. >>>Now someone has to solo Ony AND the Warden<<<

^ 3.5 rating, voted down to -4.

Impressive even with help with the warder you did solo onyxia herself and I give you props. nice job

^ 5.0 rating, voted down to -4.

...and then some from the other end of the spectrum:

DIE PLS MOTHERFUCKER

^ 1.0 rating, voted up to +2.

Grats on the kill, asshole.

^ 1.0 rating, voted up to +1.

I dont usually rate movies but I just find your attitude disgusting go die please

^ 1.0 rating, voted up to +1.

The last two pages of ratings have all their comments hidden. Positive ratings, so no surprise there. Anyway, just thought this needed to be underlined.

I guess you can either not let people comment at all, or you can let them comment and have the site turn into a flaming and bashing forum. Wish there was a third option. :S

[kvenick]

This may be a little off the current topic but I'd just like to share a report I just sent in regards to:

http://www.warcraftmovies.com/movieview.php?id=69001
The editing was the only good thing about this movie
1/1/5/1 - 2.0

The Report To Moderator:

You sit at a keyboard. I, not knowing what you do, who you are, and if you are really reading this. And if you shut down this report, there will be nothing I can do. I won't even know. But I wonder, to whom ever is reading this, what extent do you care - to what extent can you remove comments. Even so, unless I remember the people before/after the person I'm reporting, I won't even know if it worked.

You read report after report probably, bored, and without much worry whether one slips by. Call me stupid for caring, laugh at me for taking it so serious. It's not the comment or rating that bothers me, it is the person. The person that decided to try out intelligence. The person that realized he can downrate because it's so easily accessible but does so for no reason. And I wonder, who else is in that category.

What justifies whether a comment is deemed removable? Only you can answer that question. I have a video. If opinion could somehow be fact, fact would be that my video cannot be rated a 1 in any category. But what makes my opinion any greater than others? That's where your system breaks down. Where fact and opinion intertwine and the ultimate decision of right and wrong is decided upon an individual who, is just as susceptible as the next.

This man's rating is unacceptable. He acknowledges the editing (Good = Five?) but without thought, gives everything else a one. Be someone that makes the right decision this time.

[kvenick]

But to return to the subject: if you feel the current system is unfair and unbalanced, what would you suggest to improve it? Anyone can spot a problem, but to come with a working solution is what really matters. :)

Unfortunately, that's not my job. But in between free time I'd love to help, as long as I'm not the only one fixing it (sometimes you need others to point out things you forget), and I got acknowledgment. One major problem (and same for many others) I don't post a lot about balancing World of Warcraft is that they don't respond; at least in a manner where they heard you.

[kvenick]

And lets face it:the more popular WcM becomes,the more childish fanboys will drop in,the more stupid comments and senseless ratings we will see here.This goes along with a massive load of frapsed PvP "movies",wich nobody but the maker him/herself cares about.


The simplest instantaneous fix is focusing on rating weight.
But for the ultimate system, its much more complex.

It would probably involve revitalizing the entire rating system; in turn, Fame, and Rating Weight. The rating system has to be based more on fact than opinion. There are several obvious reasons but an important thing to look at is that most people look at the rating before choosing to download.

If its based on fact, it creates more realism. A description to this will also be helpful. If the new ways to rate can be established, you can move onto creating a balance of well-informed voters - stopping the up and down raters from rating or making a huge impact.

Even this won't be enough.