Saturday, August 28, 2010

Memo: HighSchoolSports.net crashed last night; outage mars debut of site's expanded coverage

The digital network serving as the backbone of a recently announced launch of more than 100 microsites devoted to high school sports failed last night, just as it was rolling out the first phase of its expanded coverage, according to an internal memo and a Gannett Blog reader.

In the memo, Peter Lundquist, vice president and general manager of the sports network, told staffers: "Although HighSchoolSports.net was restored at about 8:30 p.m. ET Friday from earlier slowness and an outage, performance degenerated to the point that the site has been difficult to use at best."

His memo continued: "This was a terrible disappointment. I apologize that we let down all of you and our customers tonight. It was devastating to me personally and to the team that has worked nonstop to make this work."

The network still appeared to be on the fritz when I checked it moments ago: Some material on the homepage wasn't displaying correctly.

Gannett announced plans to expand the network less than two weeks ago, with an initial roll out this month for 38 newspaper and TV markets. All the 100-plus sites are to be in place by the end of the year.

GCI has held a controlling stake in HighSchoolSports.net since October 2007. It's part of the company's push into niche digital sites, such as Moms Like Me, meant to supplement traditional sources of newspaper revenue.

Following is the full text of Lundquist's memo:

All,

Although HighSchoolSports.net was restored at about 8:30 p.m. ET Friday from earlier slowness and and outage, performance degenerated to the point that the site has been difficult to use at best. The engineers were not comfortable making wholesale changes that might impact stability. Instead, our developers have continued to apply changes one at a time, specifically focused on improving load times.

Our tech team will continue to work parallel tracks Saturday, one to improve page-load performance on the rolled-back code base, and the other to diagnose the root cause of the initial problem. Once we have identified and corrected the root cause to the satisfaction of the hosting team, we will re-add the improvements from this week. There are several that were not part of the rolled-back code. We will provide estimated timelines as the diagnosis is made.

Our management team is meeting first thing, and we will be providing additional communications throughout the weekend.

This was a terrible disappointment. I apologize that we let down all of you and our customers tonight. It was devastating to me personally and to the team that has worked nonstop to make this work.

-- Peter

32 comments:

  1. Oh oh.

    This high school project is a very high profile effort within the company, so stumbling out of the goal post won't be going over very well with the Red Queen:

    When logic and proportion
    Have fallen sloppy dead
    And the White Knight is talking backwards
    And the Red Queen's "off with her head!"
    Remember what the dormouse said;
    "Keep YOUR HEAD"

    ReplyDelete
  2. It is still not loading properly, and I am writing this Saturday at 10:28 am EDT.

    ReplyDelete
  3. A tragic metaphor for the rest of Gannett's fate.

    ReplyDelete
  4. After weeks of promoting this on our local platforms, giving out free flip cameras to area schools and hiring temporary workers to seed the site - now this. The site is totally frustrating to visit and navigate. Gannett has set us up for failure. Lunquist in his memo indicates they don’t have a fix, just a little spit and glue to maybe hobble along until the end of the season. One of our customers summed it up best in a comment:
    “highschoolsports.net is so slow, I could host it on my laptop and give you better performance. I'm waiting almost 120 seconds for a webpage. Fix it, or this won't be used by anyone.”

    ReplyDelete
  5. These issues have been going on for a couple of weeks in just trying to get to an individual school's schedule.

    It really sounds like an idea that was hatched and set in motion without any thought given to what it would take to make it work properly.

    Is that the traditional Gannett method of doing things?

    ReplyDelete
  6. That's funny to hear. When we called the hss.net media support line at about 6 p.m., we were told the problem was on our end and they weren't having any problems on their end.

    We've been using hss.net to some degree for the last 2 years, and it has alienated probably 90 percent of our high school coaches and readership. We went from being a pretty solid authority on high school sports to being a running joke among coaches, parents and readers. It has given our readers another reason not to pick up our paper and read our content online, which is a curious way to go as we strive for the golden local content that our consumers can't get anywhere else.

    It has not worked for what it was supposed to do from Day One, and there has been zero progress in making it better and more user-friendly. And that is despite near-daily communication to try and work with them to get this to be a viable product.

    It is manned by people who either don't know a thing about sports, don't care a thing about sports or don't have the ability to make the changes to make this "work".

    For the people in charge to even pretend that getting this fixed is a priority or to pretend that we care about high school sports coverage is a joke.

    ReplyDelete
  7. The problem has been identified: Three of the eight gerbils powering the treadmills that support the database were on furlough.

    All will be well by (Slave) Labor Day.

    On a more serious note, HSS.net is a sub-par product on an inferior platform. Either of those flaws would be an issue; together, it's a slam-dunk that something would go wrong.

    The issue is going to get worse unless they ramp up the server farm in a hurry. The rollout emphasis is on football, a once-a-week sport, at fewer than 40 sites.

    By the end of the year, they want to be doing boys and girls basketball at 100 sites, which amounts to a 10-fold increase in system stress.

    A year from now they'll want to start adding second-tier sports such as soccer, wrestling, etc. The load from both the data entry and the readership sides of the equation will be insanely higher than what caused last night's brownout.

    Good luck with that.

    ReplyDelete
  8. 1:08: In a word, NO.
    The traditional method would be to replace HighSchoolSports.net with something even buggier, and throw in a few made-up buzzwords in the announcement.

    Remember: always cover your failures with even bigger failures.

    ReplyDelete
  9. The traditional Gannett way of doing things is to pick a launch date and then figure out how to jam all the work into the time allotted. At least that's the way many projects go at my property.

    With the start of school approaching somebody probably selected the date and then it was up to the project team to get it done, whether or not it was even possible. Of course management doesn't want to hear that it can't be done even when they ask the experts doing the work how much time is needed.

    When too little time is allocated to projects that are becoming increasingly complicated something inevitably suffers. The issues that highschoolsports is suffering from may be indicative of that.

    ReplyDelete
  10. Its a blessing this sub par product never got off its insecure feet.

    A true embarassment for Gannett.

    ReplyDelete
  11. Time to use MaxPreps

    ReplyDelete
  12. Just one more company that Gannett purchased, made unrealistic expectations of, ran off the management, and then completely screwed it up.

    Lundquist once told the founder of HighSchoolSports.net that Gannett "doesn't innovate well". Looks like they execute even worse.

    Just when you thought it couldn't get any worse.... it does.

    Gannett truly does make technology impossible.

    ReplyDelete
  13. So glad we have a chief digital officer to take care of this. Looks like the stand-in, Gracia has screwed up. Maybe she will fire herself!!!

    ReplyDelete
  14. Adolph Santorine to the rescue!!! Oops, I forgot, he is suing us right now!

    ReplyDelete
  15. A typical Gannett Project.
    Shoot
    Fire
    Aim

    Why listen to those experts who know what time frame is necessary to complete successfully?

    DUH IT IS THE GANNETT WAY!

    ReplyDelete
  16. Yes, this is typical of Gannett. Rush, rush, rush. Cheap, cheap, cheap. Management pressures the workers to meet a deadline without the proper technical/financial support.

    And....While the goal line is in sight, the path to it is never clearly defined. The Gannett way is depends greatly on luck.

    That's a far cry from where this company used to be when real news people were in charge and not a bunch of finance/sales people.

    ReplyDelete
  17. What site do you work at where you're giving away free flip cameras???

    I can't even get a damn still camera to take basic photos with. A video camera? Hah! Dream on.

    ReplyDelete
  18. Still isn't working properly. Pages won't load.

    ReplyDelete
  19. We've been telling the HSS.net folks for weeks of the problems and while they are interested, they offer no solutions, and further, Mr. Lundquist is never on the calls..... Good thing football season is only a few days away......YIKES

    ReplyDelete
  20. Gannett doesn't have an idea problem. It has an execution and "gaining buy-in" problem in the ranks.

    ReplyDelete
  21. What's the surprise here? Peter Lundquist led the launch of Publicus. It crashed. Then he led the switch to Maven. It crashed. So now he's been put in charge at highschoolsports.net. What else would you expect but a crash? Best of luck to those of you in the trenches. Remember how long it took to get Publicus stable? And how long it took Gannett to give up on Maven? You can see the pattern. Get ready for a wild ride this fall.

    ReplyDelete
  22. We compete against a hss.net installation here in St. Louis. I use the term compete rather loosely.

    We saw stirrings of some type of local ramp up a couple weeks ago when the local GCI station invited coaches in to talk to them about the new and improved hss.net. It was attended by less than 20 coaches in a market of more than 200 high schools.

    http://www.stlhighschoolsports.com

    This is a site built internally and hosted locally.

    The CMS? WordPress.
    The stats database? MySQL.
    Robotic story generation? Internally coded.

    We're a staff of six who run the site, schedule the coverage, produce some content and interface with the rest of larger operation. We have eleven multimedia reporters who report up to the director of the site.

    No digital VP. No national footprint. No brownouts.

    Ironic word verification for this post: "shoot".

    ReplyDelete
  23. The HSS site has been foisted on gannett papers, forced to use this not only in place of their existing websites, but as an engine for print. The results have been predictable, particularly to anyone who's worked with it.

    The HSS back end is so poorly engineered, it should serve as a case study for an introductory level development course. Worse, it shows no insight into how news organizations actually cover sports, how high school sports are organized and run, or how a reader might wish to access HS sports information.

    Across the country, individual papers are being forced to devote countless hours to getting HSS to work for them, the end result of which will be an inferior version of what they already have in place.

    If this is how Gannett is going to move forward as a multimedia company, we're all in big trouble.

    ReplyDelete
  24. Hey stlhighschoolsports.com guy:

    Screw Gannett twice:
    Sell your site to them for six figures, and as soon as the check clears, all 6 of you QUIT!

    ReplyDelete
  25. A high school sports site should be launched in June or July - when nobody is looking at it.

    ReplyDelete
  26. Re 8:27 p.m.....What? Give yourself plenty to time to knock out the bugs? Not the Gannett way, my friend! GCI has no interest in quality. It's only interest is cost! The cheap way - which translated means minimal upfront labor - is the best way.

    Too bad the bean counters don't consider the hours that the local staffs have to spend wrestling with the latest "innovations" Then again, we're Gannett, we don't pay overtime!

    ReplyDelete
  27. @ 4:36: Given how bad GCI/hss.net seems to have screwed the pooch last weekend, the sale price for anything just went into 8 figures.

    It's not about them purchasing what they know or what they think they know.

    It's about purchasing from me what I know they don't know.

    And, it's about (to paraphrase Col. Jessep) them standing there in their faggoty business suits and with their GCI mouths extending me some fucking courtesy.

    ReplyDelete
  28. there may be some word today. (intentionally vague)

    ReplyDelete
  29. Friday night should be fun. and hopefully not as eventful as last Friday. second time's the charm? or is that 3rd?

    ReplyDelete
  30. Even when the site is working, it's nearly unusable. Who designed this thing?

    ReplyDelete
  31. Is this site ever going to work again. Been trying to upload photos for a week and all I get is the confirm your account screen.

    ReplyDelete
  32. Why would anyone use this when there's MaxPreps.com..HSS is the freshman team compared to Max Preps..Actually, maybe the pop warner team. I am forced to use HSS after using Max preps for years, and we can't stand it.

    ReplyDelete

Jim says: "Proceed with caution; this is a free-for-all comment zone. I try to correct or clarify incorrect information. But I can't catch everything. Please keep your posts focused on Gannett and media-related subjects. Note that I occasionally review comments in advance, to reject inappropriate ones. And I ignore hostile posters, and recommend you do, too."

Note: Only a member of this blog may post a comment.