Sunday, April 08, 2012

Tech 101 | Big computer, e-mail outage today

In an e-mail, a reader says: "Big computer trouble including e-mail and NewsGate across Big G today, Easter Sunday. A mess."

At 10:15 p.m. ET, The Courier-Journal in Louisville, Ky., told readers at the top of its homepage: "Due to production difficulties, the Courier-Journal will publish a combined one-edition newspaper Monday." 

78 comments:

  1. More than email is down. Couldn't logon to LAN and CCI is down.

    ReplyDelete
  2. Way to go Gannett, keep consolidating. Damn email has been messed up since last Thursday

    ReplyDelete
  3. Well, we know this won't be the last time. What a cluster.

    ReplyDelete
  4. After huge, much-watched Masters playoff, the Az Republic/AZ central website front has this Masters headline: Hanson leads, Lefty charges.

    Umm, that was yesterday, you know, before the final day's competition.

    What a disaster.

    ReplyDelete
  5. Not just Newsgate. Some of us using other systems having trouble, too. Email remains glitchy. Local guys managed a workaround at my site (amazing but true we still have a couple of IT people here) but our design hub is having trouble.

    ReplyDelete
  6. I wonder if we'll have a paper tomorrow?

    ReplyDelete
  7. I am so surprised. I feel sorry for the folks who get stuck with the failure, because most just want to do a good job.

    But the Gannett tippy-tops? These negligent, self-loving charlatans?

    They can choke on their bear-pawed idiocy.

    Sadly, though, they won't. Stinking rich.

    ReplyDelete
  8. Whoever responsible should be canned. But that never happens in Lake Wobegannett.

    ReplyDelete
  9. Sadly the "readers" will not even notice.

    ReplyDelete
  10. This is what happens when you put all of your eggs in one basket.

    ReplyDelete
  11. Worst technical failure ever. This is not Gannett Blog hype. The entire publishing system, print and online, crashed for most of Sunday. There was a chance no Gannett papers would publish at all.

    Mad scramble when things seemed to stabilize, but heads should roll. No way we should be vulnerable like that.

    ReplyDelete
    Replies
    1. Yes heads should roll, because layoffs have solved all their other problems.

      Delete
  12. My friend at one hub says they were at a standstill for hours. Way to go boys!!

    ReplyDelete
  13. We were down until 8:45 EST.

    ReplyDelete
  14. I remember in our initial meeting about the pagination hubs, and someone asking what happens in the event of a monumental system crash, is there going to be any pagination software left inhouse for emergencies.

    The head of the Design Studio laughed at that assertion and made a joke about the 12/21/12 end of the world prediction, and said there's no need to have pagination available at the individual papers with the magical Design Studio on the case. Wonder if he'd like to revisit that answer now?

    It's OK, I'm sure our readers will all forget about this when word comes down about the paywalls, price increases, etc.

    ReplyDelete
  15. What was scariest is there was no backup plan. At all.

    E-mail is still not fully restored, by the way.

    Be interesting to see how many papers were late, missed second or third editions...

    Gannett's Easter Disaster.

    ReplyDelete
    Replies
    1. Nice foresight in a backup plan.Not!

      Delete
  16. When you give Skynet total control, you get Judgment Day.

    ReplyDelete
  17. In Asbury Park, we were down until approximately 9:37. In the studio, all we could do was manually upload photos to RTC so they would be ready whenever the system came back. On the copy side, they were pulling stories off AP Exchange and editing them in Word. At about 8:30, we all started wondering what we should do if it just didn't come back. I think the answer is a simple "there's nothing we can do." Had the system not come back when it did, there's a chance there could have just been no Gannett papers tomorrow.

    Today was a complete disgrace and is incredibly embarrassing. But of course nothing will come of it and we'll continue to use this terrible system that simply does not work.

    ReplyDelete
  18. You mean the iPhones and iPads couldn't save the day? I'm shocked, shocked that Gannett had a technology failure.

    ReplyDelete
  19. All our eggs in one basket? I blame that Deal Chicken!

    ReplyDelete
  20. Contractor manifests didn't print either-it's a guess as to how many papers go to each location today.

    My email also hasn't worked since last Wed/Thurs.

    ReplyDelete
  21. Email has been off/on since thursday at the clarion-ledger. Massive computer issues today, too. What you expect from out-dated equipment, servers, etc.
    Why can't this company do anything right?

    ReplyDelete
  22. I wonder how much money the system being down cost the company in time and labor? You would think they would understand that there is a cost associated with downtime. It is a fundamental business concept. I bet the "powers to be" don't lose bonuses because of this since there isn't any accountability at that level.

    ReplyDelete
  23. Do we still have NewsGate training today?

    ReplyDelete
  24. System goes down for most of the day, but the papers got out so nothing will be done.

    Well, except to lay off more workers in the near future probably.

    ReplyDelete
  25. They will most likely plan another retreat to discuss it and cost the company thousands of dollars for useless conversation again.

    ReplyDelete
  26. Funny we were talking just a day ago that a Manager in Tech Services was leaving. gave his notice and left Friday, and before he left he sent a email to all managers saying how bad things will get now.. Joking that he should have been walked out.. we thought maybe he did something and waited to push a button and activate it.. Just a coincident?

    ReplyDelete
  27. From what I read, the email troubles started on Thursday so it is very unlikely someone in IT leaving did anything.

    ReplyDelete
  28. We start Newsgate training this week and launch at our hub later this month. How do you think that will go? We also start the paywall soon, will be transforming the newsroom, dealing with the early retirements and working on passion topics. Not too much on our plates.

    ReplyDelete
  29. We've had sporadic issues with the USA TODAY page and sharepoint for months now. Sometimes they have to e-mail us the page. I think that was the start to the slow rollout of the impending troubles.

    ReplyDelete
  30. Heck, 6:46, they'd probably make everyone work overtime and not pay them for it, telling them that "you won't get paid because you weren't able to do your job".

    ReplyDelete
  31. For what its worth. I don't know exactly what happened but I do see some inaccurate comments above.

    Ganett Corp e-mail is on up to date redundant hardware built for 100% fail over. Copies of mail are on the east and west coast. Both Coasts are backed up nightly.

    If only e-mail went down it wouldn't keep you from producing a newspaper, if only e-mail went down you would still be able to log in.

    If e-mail did go down it would mean there would be a separate catastrophic issue effecting both the east and west coast that would have effected more than email.

    Think it out kids, if everything breaks its a much larger authoritative resource that broke. Take some time, you'll figure it out.

    ReplyDelete
  32. And a hush comes over the crowd...

    ReplyDelete
  33. Spoken like an i.t. guy who has no idea how dependent this company is on email working.

    Forget about the news side - every notification that goes out from GIADC is via email. Our advertising customers depend on getting an email to tell them their proof is ready for viewing.

    We no longer have the manpower to call every customer and walk them through signing on manually and viewing their ads on ATOL, itself not exactly the most stable software.

    ReplyDelete
    Replies
    1. Exactly, we all fall down.
      Asses!!!

      Delete
  34. Bad system design is bad system design, including the lack of sufficient resources, backup and recovery. Duplicating the bad design throughout the entire company means that it is only a matter of time before multiple systems go down at once. It is a numbers game.

    ReplyDelete
  35. With the hub newspaper page production system, e-mail is more critical than ever. That's because editors can no longer shout across the room to the copy desk; much of the is done by e-mail and instant messaging.

    ReplyDelete
  36. E-mail is critical, yes, but this was a failure of the entire publishing operation. News gate was not functioning for much of the day.

    So, IT pros, how is it email brings down News gate, or was it the other way around?

    ReplyDelete
  37. Catastrophic failure. Just like this company's top management.

    ReplyDelete
  38. It wasn't email. It was active directory. This is still a problem for many sites with local editorial systems. Not due to any cuts or mismanagement.

    ReplyDelete
  39. And cuts and mismanagement had nothing to do with it taking from 2am Sunday until 10am today yesterday to get it fixed, either.

    ReplyDelete
  40. No local IT cuts had nothing to do with it. Local IT was never in the position to fix this. Go and ask your local IT person what they can do to fix active directory. As much as you would like to find a story here the one you're looking for does not exist.

    ReplyDelete
  41. Not at a Gannett paper, but I remember when we got CCI. They wanted every bit of inter-office information to be emailed. Even to a person sitting 15 feet from you. Production pretty much refused. (A few of us didn't even have company email and still don't.) Why would you let a company from the other side of the world tell you how to run your newsroom?

    ReplyDelete
  42. Local IT fixed our Arkitex system so we could send pages to plate. Without that we wouldn't have printed. Thanks, Karen!

    ReplyDelete
  43. Love to be a fly on the wall for all the Monday morning quarterbacking going on today.

    ReplyDelete
  44. Talk around our shop is that an inside sabotage job is happening. Every time layoffs or furloughs are announced, we have email issues, press problems, or some other type of "slowdown". Managers at our paper are fed up with the furloughs.

    ReplyDelete
  45. Nope that was not the case. No sabotage.

    ReplyDelete
  46. Sabotage?!?!? That's rich.
    Hello, managers? All those people you lay off? Did it ever occur to you what they do? Maybe one of them does something that keeps a particular system running and they're the only person left doing it because you laid off everyone else?
    Most of our systems don't need any sabotage. They go down on their own just fine.

    ReplyDelete
    Replies
    1. It was Aliens I tell ya!

      Delete
  47. I LOL at all of you. HAHA

    ReplyDelete
  48. Having seen previous IT projects, it would not surprise me that the Design Studios are underpowered. And local pagination software would do little good, since anyone who knew how to use it had been laid off. If it were an Active Directory problem only, the local IT staff SHOULD be able to find a way around most of it. Glad I got out when I did.

    ReplyDelete
  49. Gannett’s Mission: To lead the transformation of the media and marketing solutions industry.

    Perhaps someone can define lead as this past weekend’s events provide more proof that Gannett is doing anything but. Past experience with a few responsible for “implementing” this company’s IT efforts suggest they likely yet again failed to listen to concerns from people in the field, a management practice that explains far more problems in this company than its leadership will ever recognize and or admit.

    ReplyDelete
  50. “It’s all within reach… just now always when people want.”

    ReplyDelete
  51. Jebus, I tried to give you people a hint and you keep discussing e-mail....

    The fucking time servers went out of sync.

    AD, e-mail, CCI, that requires a logon using your AD credential didn't work because the time servers went haywire.

    Their backups are good, the infrastructure us essentially sound, the time servers got screwed up.

    The time servers are used as an authoritative resource for many applications.

    I don't know the root cause of the issue, but, it is a rarity for such an issue to happen.

    Gannett has its own time servers.

    time.gannett.com resolves to 159.54.33.78 <- Gannett owned IP.

    Shut your cock warmers about the damned e-mail. The e-mail was a symptom of a greater issue.

    ReplyDelete
  52. In an e-mail less than an hour ago, a reader wrote: "Seem 2 b dead n water again. System slowed now cant get into front end program."

    ReplyDelete
  53. I'm having no trouble at all. Nothing is slower than normal. Maybe other sites are still having local issues.

    ReplyDelete
  54. "Nothing is slower than normal?" That doesn't mean much.

    Plus I'm still getting emails that were sent to me last night.

    ReplyDelete
  55. Work slow Down...nice job...keep it going...

    ReplyDelete
  56. Are Gannett's systems in the cloud or in their own data center?

    ReplyDelete
  57. The company likely lost nothing in time or labor costs on the IT end. They're all exempt employees who get paid peanuts anyway.

    80 hour workweeks to cover for the incompetence of others was the norm during my tenure.

    ReplyDelete
  58. Just send the blank paper, I still need to wrap fish and make paper hats

    ReplyDelete
  59. The Hard working, smarter one's Do not get rewarded for Doing a Great job...all the accolades Go to the non working Management
    And there is a lot of sit Down non working management..
    Take a Furlough...Look for Better Employment..Gannett will continue to take money out of the actual workers pockets to line the upper managements pockets..
    How can you justify Doubling your income with whats going on in this Economic Enviroment..
    Spread the wealth to the people in the Trench's...give us Better Health Insurance...If not..
    Everybody SLOW DOWN...

    ReplyDelete
  60. The manager I report to was lost this morning, no emails from night shift, nothing, system was down unil at least 10am. I hear her complaining on the phone all day about furloughs, we don't have enough staffing, how he/she is tied to the job..on and on.. Must be nice to be on salary, your just a number like the rest of us.

    ReplyDelete
  61. Woah you have multiple shifts? Must be nice.

    ReplyDelete
  62. Was the meltdown at the MOC or in Phoenix? I understand that the authentication system was in total failure and thus the entire system was not accessible by some of the IT folks. Is this the result of a one time incident. Unforeseen. Or a result of poor planning and underfunding and staffing?

    ReplyDelete
  63. 4:14pm, own. The 2 major data centers are Phoenix and Maryland.

    ReplyDelete
  64. It was making the rounds at our site that Sunday's SNAFU was caused by someone putting an incorrect time stamp (2011 sted 2012) somewhere.

    ReplyDelete
  65. 11:41 - That is incorrect information. The issue was with the time servers as per above.

    ReplyDelete
  66. The "Someone". Was a computer malfunction. Are you people actually reporters or gossip columnist? No fact checking at all.

    ReplyDelete
  67. The first question I get when calling the help desk is 'what did you do?' - and it's usually snarky.

    But when IT has a problem, well that's a computer malfunction. Nothing WE did. Sometimes Skynet just does things on its own. Can't be helped. Problem at corporate. The system was set up for every contingency.

    Let me help, with the advice I always hear first - "Did you restart your PC?"

    ReplyDelete
  68. If you log a case through the website and you are a local IT person, the guys in the sky will assign it back to you despite any notes you include. If I had wanted it assigned to me, I would have done so when I created it. Round and round we go.

    ReplyDelete
  69. The Tennessean had every thing down, even printers needed re-booted, They called the Tech "can do"guy in Sunday and all he could say was " I dont know this" REALLY? Looking a bit worn and Haggard. Bite off a bit more then you could chew? All new ball game now.

    ReplyDelete
  70. @10:14 Been doing that for years..Unless I need someone to get me a Mic for a meeting.. We ALL pretty much have learned to fix it ourselves here.

    ReplyDelete
  71. IT is Great, Tech Service sucks..

    ReplyDelete
  72. 11:41 here. Unless the computers program themselves, "someone" did, indeed, play a factor in the SNAFU. The point is someone as small as an incorrect time brought a major company grinding to a halt.

    ReplyDelete
  73. 12:30 here: second "someone" should have been "something."

    ReplyDelete

Jim says: "Proceed with caution; this is a free-for-all comment zone. I try to correct or clarify incorrect information. But I can't catch everything. Please keep your posts focused on Gannett and media-related subjects. Note that I occasionally review comments in advance, to reject inappropriate ones. And I ignore hostile posters, and recommend you do, too."

Note: Only a member of this blog may post a comment.