In an e-mail, a reader says: "Big computer trouble including e-mail and NewsGate across Big G today, Easter Sunday. A mess."
At 10:15 p.m. ET, The Courier-Journal in Louisville, Ky., told readers at the top of its homepage: "Due to production difficulties, the Courier-Journal will publish a combined one-edition newspaper Monday."
At 10:15 p.m. ET, The Courier-Journal in Louisville, Ky., told readers at the top of its homepage: "Due to production difficulties, the Courier-Journal will publish a combined one-edition newspaper Monday."
More than email is down. Couldn't logon to LAN and CCI is down.
ReplyDeleteWay to go Gannett, keep consolidating. Damn email has been messed up since last Thursday
ReplyDeleteWell, we know this won't be the last time. What a cluster.
ReplyDeleteAfter huge, much-watched Masters playoff, the Az Republic/AZ central website front has this Masters headline: Hanson leads, Lefty charges.
ReplyDeleteUmm, that was yesterday, you know, before the final day's competition.
What a disaster.
Not just Newsgate. Some of us using other systems having trouble, too. Email remains glitchy. Local guys managed a workaround at my site (amazing but true we still have a couple of IT people here) but our design hub is having trouble.
ReplyDeleteI wonder if we'll have a paper tomorrow?
ReplyDeleteI am so surprised. I feel sorry for the folks who get stuck with the failure, because most just want to do a good job.
ReplyDeleteBut the Gannett tippy-tops? These negligent, self-loving charlatans?
They can choke on their bear-pawed idiocy.
Sadly, though, they won't. Stinking rich.
Whoever responsible should be canned. But that never happens in Lake Wobegannett.
ReplyDeleteSadly the "readers" will not even notice.
ReplyDeleteThis is what happens when you put all of your eggs in one basket.
ReplyDeleteWorst technical failure ever. This is not Gannett Blog hype. The entire publishing system, print and online, crashed for most of Sunday. There was a chance no Gannett papers would publish at all.
ReplyDeleteMad scramble when things seemed to stabilize, but heads should roll. No way we should be vulnerable like that.
Yes heads should roll, because layoffs have solved all their other problems.
DeleteMy friend at one hub says they were at a standstill for hours. Way to go boys!!
ReplyDeleteWe were down until 8:45 EST.
ReplyDeleteI remember in our initial meeting about the pagination hubs, and someone asking what happens in the event of a monumental system crash, is there going to be any pagination software left inhouse for emergencies.
ReplyDeleteThe head of the Design Studio laughed at that assertion and made a joke about the 12/21/12 end of the world prediction, and said there's no need to have pagination available at the individual papers with the magical Design Studio on the case. Wonder if he'd like to revisit that answer now?
It's OK, I'm sure our readers will all forget about this when word comes down about the paywalls, price increases, etc.
What was scariest is there was no backup plan. At all.
ReplyDeleteE-mail is still not fully restored, by the way.
Be interesting to see how many papers were late, missed second or third editions...
Gannett's Easter Disaster.
Nice foresight in a backup plan.Not!
DeleteWhen you give Skynet total control, you get Judgment Day.
ReplyDelete"I'll be back"
DeleteIn Asbury Park, we were down until approximately 9:37. In the studio, all we could do was manually upload photos to RTC so they would be ready whenever the system came back. On the copy side, they were pulling stories off AP Exchange and editing them in Word. At about 8:30, we all started wondering what we should do if it just didn't come back. I think the answer is a simple "there's nothing we can do." Had the system not come back when it did, there's a chance there could have just been no Gannett papers tomorrow.
ReplyDeleteToday was a complete disgrace and is incredibly embarrassing. But of course nothing will come of it and we'll continue to use this terrible system that simply does not work.
You mean the iPhones and iPads couldn't save the day? I'm shocked, shocked that Gannett had a technology failure.
ReplyDeleteAll our eggs in one basket? I blame that Deal Chicken!
ReplyDeleteContractor manifests didn't print either-it's a guess as to how many papers go to each location today.
ReplyDeleteMy email also hasn't worked since last Wed/Thurs.
Email has been off/on since thursday at the clarion-ledger. Massive computer issues today, too. What you expect from out-dated equipment, servers, etc.
ReplyDeleteWhy can't this company do anything right?
I wonder how much money the system being down cost the company in time and labor? You would think they would understand that there is a cost associated with downtime. It is a fundamental business concept. I bet the "powers to be" don't lose bonuses because of this since there isn't any accountability at that level.
ReplyDeleteDo we still have NewsGate training today?
ReplyDeleteSystem goes down for most of the day, but the papers got out so nothing will be done.
ReplyDeleteWell, except to lay off more workers in the near future probably.
They will most likely plan another retreat to discuss it and cost the company thousands of dollars for useless conversation again.
ReplyDeleteFunny we were talking just a day ago that a Manager in Tech Services was leaving. gave his notice and left Friday, and before he left he sent a email to all managers saying how bad things will get now.. Joking that he should have been walked out.. we thought maybe he did something and waited to push a button and activate it.. Just a coincident?
ReplyDeleteFrom what I read, the email troubles started on Thursday so it is very unlikely someone in IT leaving did anything.
ReplyDeleteWe start Newsgate training this week and launch at our hub later this month. How do you think that will go? We also start the paywall soon, will be transforming the newsroom, dealing with the early retirements and working on passion topics. Not too much on our plates.
ReplyDeleteWe've had sporadic issues with the USA TODAY page and sharepoint for months now. Sometimes they have to e-mail us the page. I think that was the start to the slow rollout of the impending troubles.
ReplyDeleteHeck, 6:46, they'd probably make everyone work overtime and not pay them for it, telling them that "you won't get paid because you weren't able to do your job".
ReplyDeleteFor what its worth. I don't know exactly what happened but I do see some inaccurate comments above.
ReplyDeleteGanett Corp e-mail is on up to date redundant hardware built for 100% fail over. Copies of mail are on the east and west coast. Both Coasts are backed up nightly.
If only e-mail went down it wouldn't keep you from producing a newspaper, if only e-mail went down you would still be able to log in.
If e-mail did go down it would mean there would be a separate catastrophic issue effecting both the east and west coast that would have effected more than email.
Think it out kids, if everything breaks its a much larger authoritative resource that broke. Take some time, you'll figure it out.
And a hush comes over the crowd...
ReplyDeleteSpoken like an i.t. guy who has no idea how dependent this company is on email working.
ReplyDeleteForget about the news side - every notification that goes out from GIADC is via email. Our advertising customers depend on getting an email to tell them their proof is ready for viewing.
We no longer have the manpower to call every customer and walk them through signing on manually and viewing their ads on ATOL, itself not exactly the most stable software.
Exactly, we all fall down.
DeleteAsses!!!
Bad system design is bad system design, including the lack of sufficient resources, backup and recovery. Duplicating the bad design throughout the entire company means that it is only a matter of time before multiple systems go down at once. It is a numbers game.
ReplyDeleteWith the hub newspaper page production system, e-mail is more critical than ever. That's because editors can no longer shout across the room to the copy desk; much of the is done by e-mail and instant messaging.
ReplyDeleteE-mail is critical, yes, but this was a failure of the entire publishing operation. News gate was not functioning for much of the day.
ReplyDeleteSo, IT pros, how is it email brings down News gate, or was it the other way around?
Catastrophic failure. Just like this company's top management.
ReplyDeleteIt wasn't email. It was active directory. This is still a problem for many sites with local editorial systems. Not due to any cuts or mismanagement.
ReplyDeleteAnd cuts and mismanagement had nothing to do with it taking from 2am Sunday until 10am today yesterday to get it fixed, either.
ReplyDeleteNo local IT cuts had nothing to do with it. Local IT was never in the position to fix this. Go and ask your local IT person what they can do to fix active directory. As much as you would like to find a story here the one you're looking for does not exist.
ReplyDeleteNot at a Gannett paper, but I remember when we got CCI. They wanted every bit of inter-office information to be emailed. Even to a person sitting 15 feet from you. Production pretty much refused. (A few of us didn't even have company email and still don't.) Why would you let a company from the other side of the world tell you how to run your newsroom?
ReplyDeleteLocal IT fixed our Arkitex system so we could send pages to plate. Without that we wouldn't have printed. Thanks, Karen!
ReplyDeleteLove to be a fly on the wall for all the Monday morning quarterbacking going on today.
ReplyDeleteTalk around our shop is that an inside sabotage job is happening. Every time layoffs or furloughs are announced, we have email issues, press problems, or some other type of "slowdown". Managers at our paper are fed up with the furloughs.
ReplyDeleteNope that was not the case. No sabotage.
ReplyDeleteSabotage?!?!? That's rich.
ReplyDeleteHello, managers? All those people you lay off? Did it ever occur to you what they do? Maybe one of them does something that keeps a particular system running and they're the only person left doing it because you laid off everyone else?
Most of our systems don't need any sabotage. They go down on their own just fine.
It was Aliens I tell ya!
DeleteI LOL at all of you. HAHA
ReplyDeleteHaving seen previous IT projects, it would not surprise me that the Design Studios are underpowered. And local pagination software would do little good, since anyone who knew how to use it had been laid off. If it were an Active Directory problem only, the local IT staff SHOULD be able to find a way around most of it. Glad I got out when I did.
ReplyDeleteGannett’s Mission: To lead the transformation of the media and marketing solutions industry.
ReplyDeletePerhaps someone can define lead as this past weekend’s events provide more proof that Gannett is doing anything but. Past experience with a few responsible for “implementing” this company’s IT efforts suggest they likely yet again failed to listen to concerns from people in the field, a management practice that explains far more problems in this company than its leadership will ever recognize and or admit.
“It’s all within reach… just now always when people want.”
ReplyDeleteJebus, I tried to give you people a hint and you keep discussing e-mail....
ReplyDeleteThe fucking time servers went out of sync.
AD, e-mail, CCI, that requires a logon using your AD credential didn't work because the time servers went haywire.
Their backups are good, the infrastructure us essentially sound, the time servers got screwed up.
The time servers are used as an authoritative resource for many applications.
I don't know the root cause of the issue, but, it is a rarity for such an issue to happen.
Gannett has its own time servers.
time.gannett.com resolves to 159.54.33.78 <- Gannett owned IP.
Shut your cock warmers about the damned e-mail. The e-mail was a symptom of a greater issue.
In an e-mail less than an hour ago, a reader wrote: "Seem 2 b dead n water again. System slowed now cant get into front end program."
ReplyDeleteI'm having no trouble at all. Nothing is slower than normal. Maybe other sites are still having local issues.
ReplyDelete"Nothing is slower than normal?" That doesn't mean much.
ReplyDeletePlus I'm still getting emails that were sent to me last night.
Work slow Down...nice job...keep it going...
ReplyDeleteAre Gannett's systems in the cloud or in their own data center?
ReplyDeleteThe company likely lost nothing in time or labor costs on the IT end. They're all exempt employees who get paid peanuts anyway.
ReplyDelete80 hour workweeks to cover for the incompetence of others was the norm during my tenure.
Just send the blank paper, I still need to wrap fish and make paper hats
ReplyDeleteThe Hard working, smarter one's Do not get rewarded for Doing a Great job...all the accolades Go to the non working Management
ReplyDeleteAnd there is a lot of sit Down non working management..
Take a Furlough...Look for Better Employment..Gannett will continue to take money out of the actual workers pockets to line the upper managements pockets..
How can you justify Doubling your income with whats going on in this Economic Enviroment..
Spread the wealth to the people in the Trench's...give us Better Health Insurance...If not..
Everybody SLOW DOWN...
The manager I report to was lost this morning, no emails from night shift, nothing, system was down unil at least 10am. I hear her complaining on the phone all day about furloughs, we don't have enough staffing, how he/she is tied to the job..on and on.. Must be nice to be on salary, your just a number like the rest of us.
ReplyDeleteWoah you have multiple shifts? Must be nice.
ReplyDeleteWas the meltdown at the MOC or in Phoenix? I understand that the authentication system was in total failure and thus the entire system was not accessible by some of the IT folks. Is this the result of a one time incident. Unforeseen. Or a result of poor planning and underfunding and staffing?
ReplyDelete4:14pm, own. The 2 major data centers are Phoenix and Maryland.
ReplyDeleteIt was making the rounds at our site that Sunday's SNAFU was caused by someone putting an incorrect time stamp (2011 sted 2012) somewhere.
ReplyDelete11:41 - That is incorrect information. The issue was with the time servers as per above.
ReplyDeleteThe "Someone". Was a computer malfunction. Are you people actually reporters or gossip columnist? No fact checking at all.
ReplyDeleteThe first question I get when calling the help desk is 'what did you do?' - and it's usually snarky.
ReplyDeleteBut when IT has a problem, well that's a computer malfunction. Nothing WE did. Sometimes Skynet just does things on its own. Can't be helped. Problem at corporate. The system was set up for every contingency.
Let me help, with the advice I always hear first - "Did you restart your PC?"
If you log a case through the website and you are a local IT person, the guys in the sky will assign it back to you despite any notes you include. If I had wanted it assigned to me, I would have done so when I created it. Round and round we go.
ReplyDeleteThe Tennessean had every thing down, even printers needed re-booted, They called the Tech "can do"guy in Sunday and all he could say was " I dont know this" REALLY? Looking a bit worn and Haggard. Bite off a bit more then you could chew? All new ball game now.
ReplyDelete@10:14 Been doing that for years..Unless I need someone to get me a Mic for a meeting.. We ALL pretty much have learned to fix it ourselves here.
ReplyDeleteIT is Great, Tech Service sucks..
ReplyDelete11:41 here. Unless the computers program themselves, "someone" did, indeed, play a factor in the SNAFU. The point is someone as small as an incorrect time brought a major company grinding to a halt.
ReplyDelete12:30 here: second "someone" should have been "something."
ReplyDelete