Let’s Talk About Boards Moderation

RiotKeyru·7/16/2015, 11:19:40 PM·34 votes·24,128 views

Hey guys!

Keyru here to discuss moderation on the NA boards, including both what we’re currently working on and our long-term vision. A lot of you have provided really good feedback about moderation as part of our wider discussion regarding the updates coming to the NA boards Part 1, Part 2, and Part 3. Hopefully I can provide some insight into how we look at moderation as well as where it’s going.

First, for clarity, as many of you may know we partner with a third-party company called Metaverse in moderating the boards. I’m actually responsible for that relationship these days, and have been working with Metaverse closely on to make sure we’re aligned on what’s working and what needs to improve.

One of the major points of feedback we’ve heard from you guys concerns the level of transparency and visibility around moderation on the Boards, so that’s obviously been a big part of both our internal planning and my latest communication and alignment efforts with Metaverse. Ultimately, we’d like both the specifics of our moderation policies and the mechanics of how they’re implemented to be more clear to you guys. To that end, I’d like to provide some info about changes we’re planning to shortly make to moderation guidelines and processes on the boards. Here’s the current list:

[FUTURE CHANGE] Modify the Universal Rules to be clearer and on-point, which should improve consistency for both players and moderators. For example, here’s a snippet from the current draft of the updated Universal Rules:

Spamming / Trolling

Here is a breakdown of the various types of spam / trolling that the moderators look for and will take action upon:

  • Posting content not related to the discussion
  • Posting designed to bait conversation or falsify content
  • Making repetitive non-constructive or low effort content
  • Bumping of discussions with no additional context
  • Excessive emoji / icon spam

[FUTURE CHANGE] Concurrently with the updated Universal Rules, begin holding Boards posters more closely to the letter of the rules and be more aggressive when it comes to enforcing them (no more 50+ violations without being banned). We’re diligently working to ensure the updated rules are easy to understand, comprehensive, and representative of common values shared among members of the League community. Ideally, the rules should be simple to follow, and the lines clear - when they are crossed, appropriate action should swiftly follow.

[IMMEDIATE CHANGE] Institute a blanket policy across all sub-boards, including GD, disallowing conversations regarding religion, race, politics, sex, and social issue topics unrelated to League of Legends (i.e. online legislation is ok, fracking legislation is not). This is an interim step to aid adjustment as we move towards eventual new Universal Rules, sub-board guidelines, and moderation policies as a part of the overall boards updates.

[IMMEDIATE CHANGE] Update moderation notifications to more accurately inform players of both the reason for their punishment / warning and the consequences. While deleted comments aren’t visible, mod messages will include both a copy-paste of the actionable comment and a direct URL for the discussion in which the comment was posted. Here’s an example of a message a player would receive for using a racist, homophobic, or sexist term:

Hey XXX,

We here at Riot do not condone the use of racial, homophobic, or sexist slurs. You have been given a final warning and 7 day suspension. Please be more mindful in the future of those around you.

Here is your comment:

[QUOTE]

And a link to the thread in which it was posted:

[LINK]

[IMMEDIATE CHANGE] Going forward, we’ll be replying and answering in more moderation based discussions, about general policies or specific actions that have been taken (myself and HurriKane aka Moderator Kane will fill this role for the time being). This should help with transparency and let players know our thought process on why we take certain actions. Long-term, we’ll be looking to give these conversations a more specific sub-board home (along with wider Player Behavior conversations) as part of the upcoming boards reorganization.

We also continue to work with the Boards dev team when it comes to prioritizing features, and the same goes for moderation related ones. Some of the things we’re interested in developing (no promises) include:

  • Moderation appeals
  • Ability to respond directly to moderation notifications
  • More report options and an open text field
  • Personal Boards blacklist/muting feature
  • Ability to move discussions instead of deleting them, where appropriate

Of course, let us know which of the features you feel are most important as well as any other mod-related ones you’d like to see.

As mentioned in the bullets, we’re pushing a couple of these changes live immediately (i.e. on Monday, 7/20). Us hopping into more moderation discussions and better notifications are all upside, while the interim blanket rule against particularly unrelated discussions should mostly be a light corrective measure to the worst excesses and help ease transition towards the eventual new Universal Rules. Obviously, this last change will most significantly affect GD, where the updated rule will be reflected in an addition to the sub-board guidelines pending new Universal Rules.

Going forward, we’ll continue to develop updated Universal Rules and of course share them with you guys for discussion. Further down the line, the Player Relations guys are planning to kick off a discussion around our (currently still vague) ideas regarding community mods / MVPs. We’d like to explore with you guys how that kind of system could potentially enable players to feel more ownership over their Boards and their moderation.

The moderators, myself, and the rest of the NA boards team would love your feedback, so feel free to share any thoughts you may have on moderation!

137 Comments

Norscout7/17/2015, 5:07:57 AM15 votes

The problem I have with that rule "low effort" post, is...it just seems like an arbitrary rule that can be punished based on whatever mood the mod is in. Not every one agrees with what considered a "shit post" and tbh, I find a lot of em funny as hell. Seems like a goofy rule to have on a video game General Discussion board.

ModIts Yuu and Mi7/16/2015, 11:42:07 PM8 votes

Hmm.

A few thoughts.

First off, thank you for -finally- openly talking about the 3rd party mods. If you guys honestly think that's best for the community, fine, it just felt like it was hidden in secret which causes a lot of trust issues. I think the whole "clarity" aspect is really needed and healthy. People are more willing to be understanding if things are 100% clear, so thank you for this.

Second,

Spamming / Trolling Here is a breakdown of the various types of spam / trolling that the moderators look for and will take action upon: Posting content not related to the discussion Posting designed to bait conversation or falsify content Making repetitive non-constructive or low effort content Bumping of discussions with no additional context Excessive emoji / icon spam

Only thing I wish to comment here is a LOT of GD'ers, and to be honest myself included, are really afraid of what "low content" means. We're still aggravated that GD is being removed, fine, whatever (not happy lol), but we're still a community and we all want "off topic" posts to remain. I don't, and we don't, think it's fair for any chance of that to be removed. I don't know what you guys have planned for that direction, but we're not going to be happy if things end up that way and that'll lead to a lot of bad directions. History tends to repeat itself.

Enough said on that, onward!

[IMMEDIATE CHANGE] Update moderation notifications to more accurately inform players of both the reason for their punishment / warning and the consequences. While deleted comments aren’t visible, mod messages will include both a copy-paste of the actionable comment and a direct URL for the discussion in which the comment was posted. Here’s an example of a message a player would receive for using a racist, homophobic, or sexist term:

Thank you, for the love of fucking god thank you. About time.

Next:

We also continue to work with the Boards dev team when it comes to prioritizing features, and the same goes for moderation related ones. Some of the things we’re interested in developing (no promises) include:

Moderation appeals Ability to respond directly to moderation notifications More report options and an open text field Personal Boards blacklist/muting feature Ability to move discussions instead of deleting them, where appropriate

So, I guess a few things. First off, I don't see why this is "future but unsure". A "report with an open text box" is uh... Standard on like, every single website ever. Why should this be any different? It doesn't take long to code (seriously, do it in a lunch break), and it makes things easier. What's the downside? Are you like, afraid someone's going to be a child with it and just write up some stupid things or what... I mean honestly this is going to do a lot more good than harm, I have no idea why this is on the "maybe" list. It could be done in less than a week really easily. I don't know why you guys insist on being so slow on this.

Personal blacklist/muting feature: Same thing. If we can mute in game we should be able to do that here. Actually, everyone with FEK does. Why is a 3rd party doing something you guys should? Just being blunt and honest-- not trying to be a jerk but constructive; I know this is going to come off harsh.

Unsure on the moving discussions but we'll see...

And finally, the moderator appeals I understand why that's a maybe for a hundred different reasons, but I'm glad you guys are considering them.


I suppose that's all I have to say, so thank you for your time.

Best Furry NA/Valiant Sentinel

Starlighte7/16/2015, 11:45:57 PM7 votes

I'm going to touch briefly on my knee-jerk reaction on this and then give it some thought and revisit the thread.

The changes to the universal rules are a very pointed attack at GD and GD culture. Visible moderation is great, as are moderator appeals, but what isn't great is how vague you're being about "visible moderators" concerns me.

Give me some time to have a more well thought-out opinion. As it stands though, this is 50-50 the right direction.

Daen7/16/2015, 11:42:56 PM6 votes

Preface, thank you. This thread is really well written and it's pretty clear you have been listening to our concerns when it comes to moderation, so thank you. I'm looking forward to a lot of what you've mentioned, and I hope you continue to include us in discussions like these moving forward.


Here is a breakdown of the various types of spam / trolling that the moderators look for and will take action upon:

  • Posting designed to bait conversation or falsify content
  • Bumping of discussions with no additional context

Yes please. The other things on this list (the ones I didn't quote) don't feel like huge problems to me right now, but I am curious as to how the moderation team plans to define "low effort", from a moderation standpoint.


Concurrently with the updated Universal Rules, begin holding Boards posters more closely to the letter of the rules and be more aggressive when it comes to enforcing them (no more 50+ violations without being banned). We’re diligently working to ensure the updated rules are easy to understand, comprehensive, and representative of common values shared among members of the League community. Ideally, the rules should be simple to follow, and the lines clear - when they are crossed, appropriate action should swiftly follow.

This is both refreshing to see and moderately concerning, depending on the intent behind it. Throughout the Boards' lifespan I've always felt as though the very first punishment or two doesn't necessarily improve the community or match the action (e.g. a new player posting in the wrong place getting his or her content permanently removed), so I'm hoping this applies more to clear breaches of conduct rather than accidents or slight overstepping.

Institute a blanket policy across all sub-boards, including GD, disallowing conversations regarding religion, race, politics, sex, and social issue topics unrelated to League of Legends

This one sounds a bit heavy handed, though my assumption is that it wouldn't necessarily have to be? As an example, what action would you expect the moderation team to take against a thread that's talking about social unrest in <insert foreign nation here> and is generally constructive?

Actually, I see lower in the thread you mentioned the following:

" the interim blanket rule against particularly unrelated discussions should mostly be a light corrective measure to the worst excesses and help ease transition towards the eventual new Universal Rules"

I think that answers my question.


Update moderation notifications to more accurately inform players of both the reason for their punishment / warning and the consequences. While deleted comments aren’t visible, mod messages will include both a copy-paste of the actionable comment and a direct URL for the discussion in which the comment was posted.

This is a huge deal, I look forward to this change!

we’ll be replying and answering in more moderation based discussions, about general policies or specific actions that have been taken (myself and HurriKane aka Moderator Kane will fill this role for the time being).

In the short term, will this type of content be based in the Boards Feedback board or just kinda pop up wherever it happens?


Moderation appeals Ability to respond directly to moderation notifications

These would be great.

More report options and an open text field

Is there a reason this is a future plan rather than a relatively immediate change? I wouldn't expect something like this to be difficult or time consuming to implement, but if I'm wrong I'd love to understand why.

Personal Boards blacklist/muting feature

I'm not actually sure whether this would be a positive or a negative overall, since ideally moderation should be handling the vast majority of content we as participants might want to remove. What reasons do you see for adding this feature down the line?

Ability to move discussions instead of deleting them, where appropriate

Yessssss this would be an incredibly good change, especially for new users that get a little confused on their first couple posts.


Further down the line, the Player Relations guys are planning to kick off a discussion around our (currently still vague) ideas regarding community mods / MVPs. We’d like to explore with you guys how that kind of system could potentially enable players to feel more ownership over their Boards and their moderation.

Very much looking forward to this, as I'm sure many others are as well. Thanks for letting us know what may be in the pipeline for later! Do you happen to have a super rough timeline on when that discussion might be (a month, six decades, etc.)?

Easyaeta7/17/2015, 2:59:23 AM5 votes

Moderation here is more disgusting than the last girl I slept with

Starlighte7/17/2015, 1:54:57 AM5 votes

So here goes:

Your changes to the Universal Rules without a doubt, make them clearer. But is that necessarily a good thing? You may believe so considering that this is what Riot has decided on, but I think the community would disagree if these were fully implemented and enforced by mods. How much wiggle room does this plan have going forward? You've mentioned that this is a conversation, or is this a statement of "This is what we are doing, what do you guys think (We'll read the feedback, and say we've heard you, but probably not change anything.)? Because let's be honest, a LOT of the changes on the boards have been presented that way, and there is a current track record that makes it difficult to trust that there is going to be some honest conversations happening between the Boards Dev Team and the community.

Speaking of community, these proposed changes (parts 1-3) not only remove GD, but destroys many aspects of it's culture. Now, Keyru, you, d'art, and the rest of the team may be at a stage of utter disagreement with me when it comes to the value of GD. I personally believe that there is no reason to remove that culture, and doing so only alienates one of the loudest and most loyal portions of your fanbase. I'm not convinced that this is healthy. Perhaps long-term for the Boards it is healthy, but it could be argued that long-term health would be in condensing communication: instead of Twitter and Reddit and Facebook and Boards, having one centralized location is FAR more beneficial to the long-term health of the Boards, compared to removing GD.

Moderation. As I mentioned earlier, the fact that you don't even mention visible moderators makes it sound as if that is entirely off the table. Why? Let's talk about the pros and cons about visible moderation, and moderator accountability. As we've seen over the past few months, mods have been neither better nor worse when it comes to "Not removing content that shouldn't be removed." but what HAS been better in that regard is Riot's response to improperly removed content. Visible Moderation Action is important, especially when we're talking about long-term platform health. Not just making the communication with the person who has had action taken against their account (that should be a no-brainer, we all agree!) but making the action available to the greater community. This provides visibility to proper (and improper) actions taken. No more witch-hunts. And no more invisible mods. These are absolute musts, and any conversation about moderation changes MUST revolve around them.

And here is the hardest part of all. We need a concrete timeline. We get it, things get pushed-back. Then communicate those changes needing additional time, and reasons as to why if you can. But regardless, we can NOT continue down this path of updates seemingly haphazardly and without messaging (i.e. when a specific 4-letter word was suddenly a word worthy of banning and censorship) these things are damaging to the community, especially with the "new" relationship the community has with the Boards Team since it has undergone massive changes. You have to commit to communicating changes that are being worked on, and tentative timelines as to "when you can expect this to be a serious possibility" because even gameplay and balance decisions get time to sit on the PBE and ferment.

Lastly, I applaud the focus on making League forums related to LoL. At the end of the day it is easier to moderate. I think it is a very... worrisome position, to be forbidding a vast array of topics from being spoken about. I hope that Riot as a company worries more about their community, than their public perception, because in this particular series of changes, I fear that to be an accurate representation of the current mindset within Riot.

Jesus is Savior7/16/2015, 11:30:24 PM4 votes

first!

*deleted for spam*

:(

elduris7/16/2015, 11:44:10 PM4 votes

Hey Keyru, great post!

I'd be behind the community mods idea. Select a few of the more dedicated members from each sub board and allow them to have minor mod powers in the ability to sticky or close threads and move threads to another board. It doesn't need to be super comprehensive like being able to issue warnings or ban people from the boards, but these minor tools will come in quite handy. As someone who frequents Help & Support, very often I wish I could create a sticky thread for emerging issues or move a thread to a board that it actually belongs to.

At any rate, I'm happy about this added transparency! Being more clear on mod actions across the board can only ever be a good thing for Riot and the community both. Looking forward to seeing these changes implemented!

Mr Broken Wings7/21/2015, 12:39:24 PM3 votes

The mods currently delete "low effort" threads in the gd forum. "This is the wild wild west. Unlike the rest of the boards, anything goes. " This forum is for conversations like this and having fun, could you guys please stop overlooking your culture descriptions? Thank you. Other then that, your moderation is random and people always feel like you just delete some posts because you are bored or something, consistency is something you guys need to work on A LOT.

LittleBlast7/17/2015, 1:20:30 AM3 votes

Can we have it so that there wont be like 30 nerf riven threads at a time?

ModIts Yuu and Mi7/17/2015, 5:09:32 PM3 votes

Oh, and I should mention...

Those who discuss the "purge of GD", and this time I actually am not referencing any one player, you should be aware of something.

Most of us have been around for years, on GD. If you think we're leaving, most of us aren't. While we may frustratingly adapt to certain rules, we're not going to change as people.

So that means, we're not vanishing. We're not just being locked away. We're not disappearing in smoke.

I've suggested it to one person, but I suggest you post in a few threads on GD, and make a couple of your own. I'd advise you to stray away from anything that specifically mentions you as an "outsider" because that can be interpreted the wrong way. Check us out, and see what kinds of people we are.

Yes, some of us aren't nice. Welcome to people -everywhere-. But a lot of us are either kind, or neutral.

We're going to be with you if the changes do go through and we get mixed together. We'll be posting in the same sections as you, and you'll have to see us.

I'd advise you to not have a bitter mindset about us, because that'll just bite you in the ass.

imonly4noobcakes7/18/2015, 2:28:40 PM3 votes

So you will stop trolls in forums but are doing ZERO for trolls in ranked??? WHAT??? Please fix the troll problem in the game. I"m so sick of trolls...that I am playing less than I used to, and I've stopped investing in this game, because lets face it why should I spend hard earned money when some troll kid who doesn't even use his own money is trolling the shit out of games every day.

C9 Scott Free 7/20/2015, 3:55:49 PM3 votes

At first I was mad because we wont be able to discuss philosophical things any more. Then I realized no more Faux Feminist posts and decided it was all ok.

Tundra Fizz7/17/2015, 2:51:04 AM3 votes

Hey Keyru, is it possible for us to have warnings removed from our record if we go long enough without getting another one, or fulfill some other circumstance?

I ask this because I have exactly one warning that's from three months ago, and it doesn't explain anything about what was removed. It's really annoying having that one single warning just sit in there. If you have the time to answer this, thanks.

http://puu.sh/j24Wt.png

Enosetc7/17/2015, 2:43:25 AM2 votes

{quoted}

[IMMEDIATE CHANGE] Institute a blanket policy across all sub-boards, including GD, disallowing conversations regarding religion, race, politics, sex, and social issue topics unrelated to League of Legends (i.e. online legislation is ok, fracking legislation is not). This is an interim step to aid adjustment as we move towards eventual new Universal Rules, sub-board guidelines, and moderation policies as a part of the overall boards updates.

[IMMEDIATE CHANGE] Update moderation notifications to more accurately inform players of both the reason for their punishment / warning and the consequences. While deleted comments aren’t visible, mod messages will include both a copy-paste of the actionable comment and a direct URL for the discussion in which the comment was posted. Here’s an example of a message a player would receive for using a racist, homophobic, or sexist term:

[IMMEDIATE CHANGE] Going forward, we’ll be replying and answering in more moderation based discussions, about general policies or specific actions that have been taken (myself and HurriKane aka Moderator Kane will fill this role for the time being). This should help with transparency and let players know our thought process on why we take certain actions. Long-term, we’ll be looking to give these conversations a more specific sub-board home (along with wider Player Behavior conversations) as part of the upcoming boards reorganization.

You have no idea how happy I am to read this -- especially the first immediate change.

I was expecting to come in this thread with an essay-long response on what I feel we haven't quite gotten through to Riot on, but these changes that you've listed have all but shut me up.

One thing I want to emphasize is the importance of staying consistent when these changes go into full effect. Far too often does Riot say "change gon' come", follow up with said change and then slowly fall off when things don't quite work out as planned initially. I've already seen a lot of the immediate changes you've listed and I'm elated that the changes to how moderation works have come into fruition.

Thanks for listening to our feedback and also for taking some time to take part in the discussions going on in our favorite "cesspool". It's much appreciated.

(also, HI KANE)

Vacus7/17/2015, 2:47:09 AM2 votes

I'm generally a fan of these things.

As BFNA noted, my biggest quibble is that some low-effort content or deviations from a strict topic are hilarious and something that we want to preserve. And of course, sometimes the topic of conversation can morph over the course of the thread, and I don't think there's anything wrong with that.

Basically, I think these rules need to be applied as guidelines envisioned as a wave, rather than a line. Imagine a sine or cosine wave, going back and forth around a central path; that's a fine topic as long as no one strays way too far off. A straight line is quite useful for some forums or topics (Help and Support comes to mind; no one wants to read jokes when they're asking for tech help, those threads need to stay laser-focused on topic), but for a place where someone is actually coming to relax such as GD or whatever replaces it, they're perfectly acceptable and should even be encouraged in my opinion, because these things are how a community grows and stays.

I am a big fan of the blanket ban on controversial topics that have no relation to LoL, though; they offer no real value and are roughly 99% trolls.

I'm going to ignore all the upcoming features because no offense, you're not making any promises, I will have an opinion on them when there's a more firm commitment to actually improving them. The Boards development in general has not impressed me, neither in organization nor in design, and I choose not to sign up for further disappointment by actually expecting new useful features.

Social Justice 17/17/2015, 3:02:20 AM2 votes

Could you add transphobic and ableist slurs to the list of things Boards doesn't tolerate?

disregardable7/17/2015, 4:20:18 AM2 votes
  1. Glad a personal block list is at least something you're thinking about. I feel heard!
  2. Agree that moderators need their visibility stepped up. As it is now, they really do feel like bots*.
  3. I'm really concerned that the important features are just listed as far-off "no-promises". I mean, other than punishing the GD and making the mods copy and paste, this doesn't sound like there any concrete goals to improve boards moderation.
  4. Lastly, I know the goal is to not have people skirt around the rules, but I'd say that having ill-defined rules actually encourages that. There have been MANY examples I've seen where if they were on another forum with similar "don't be a jerk" rules as this one, it would have been deleted in ~15 minutes, but because it was here it was deemed not worth deleting.

As for community mods, please just don't.

*omg autocorrect please

Deep Terror Nami7/17/2015, 1:35:54 AM1 votes

Thank you for adding the trolling policy. Without that being an actual rule, they just run around and shit all over the place currently. After a few times of getting banned or their content deleted they learn where the line is that they can dance on without breaking the rules, but still being equally as annoying.

One thing I'd like to see changed for Spamming is what is considered spam. Currently the rules literally state that you can post the same thing on every relevant board. Posting it 4 times on one board is effectively no different from posting it 4 times on four different but arguably appropriate boards (it's always Misc, League videos, community creations, and Memes & Games) because when you look at the Boards front page it shows every one of them. There's no difference, it just shows the same post 4 times in a row.

tooomine7/28/2015, 2:07:11 AM1 votes

Cool.

Yo, your initial post leave me kind of leary about posting at all. I'm all for having areas where its not ok to touch on hot button issues, but there needs to be a part of the boards where anything, including religion, racial slurs, politics, sex, and social topics unrelated to league of legends are allowed.

I'd like to see mods from the community. I'd like to be able to vote mods in or out based on their performance, too.

if this is going to be a community at all, and if that community is going to have any worth as a tool from which riot can glean insights into relevant design philosophies they are not currently implementing, starting with the categorical exclusion of ideas doesn't make for an honest and responsive environment. you probably want to rethink that, or you might find that in 2 years, your game is doing super well according to your boards, but no one is playing it anymore becuase you wouldn't listen to people who had different opinions about topics that they weren't allowed to discuss without being banned.

You should probably ask yourself "what do we, as riot games, really want for our boards? a place where we can look over feedback that is real, and our player base feels safe giving that feedback to us, or a place where the topics are all nice topics, and anyone who has a dissenting opinion about our recent champion updates gets silenced because they "baited a discussion", or "didn't provide an additional context", or it was a relatively "low effort" post".

You want to enforce those ideas, that's fine, but when the abuse comes out, as a system will inevitably have at some point, having a space of the board relatively unmoderated will serve as a "pressure release valve" for this system that you're trying to impolement. if you place these strictures in place across the whole board, its quite possible that you'll lose the whole worth of the system.

PaladinNO7/30/2015, 6:41:38 PM1 votes

Will this policy on moderation and the moderation itself be applied to the news page comment section as well? Quite tired of seeing 50% of the posts being scam attempts of various sorts, and additional 40% being off topic there.

Example: http://eune.leagueoflegends.com/en/news/store/sales/champion-and-skin-sale-3107-0308

Lord Allah7/30/2015, 11:25:42 PM1 votes

fghrh

Soulsie8/4/2015, 9:54:53 PM1 votes

pls bring back GD

Chymaerus8/6/2015, 9:22:16 PM1 votes

I'm kind of mind blown at the way these rules seem to be enforced. I've never posted a troll comment before (troll being a comment intended to incite or goad someone to emotional distress and meaningless debate) nor said anything inflammatory; I've never derailed a serious discussion or meaninglessly bumped anyone's comments, never used an icon or emote. I make posts infrequently, and yet I've noticed that about half of the very lighthearted jovial remarks I've made mysteriously vanish from the boards moments after I post them, even when they're in line with comments that came before.

It makes me seriously wonder what in the world you guys are doing and whats going on in your minds...if you want 0 humor of any kind on your boards you should probably make a high profile news announcement about it, because that genuinely seems to be the standard you are shooting for based on what I'm seeing.