Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…
Funny thing is this kind of behaviour isn’t unique to Musk. A lot of entrepreneurs and CEOs seem to have similar kind of attitude. They want everything done cheaper faster and there’s no 2 ways about it. It’s their way or highway. If shit goes to hell it’s other people’s heads that roll.
This is exactly why our last governor here in Illinois, didn’t CEO Governor Rauner, was probably the most ineffectual governor of our state ever had. He literally had no capacity for compromise, and was a Republican, in an Illinois where the legislature has long been solidly Democrat. I can’t think of a single thing he actually got done. All because he was a CEO trying to run Illinois like a business.
I do not recommend working in Compliance then either lol!
It’s frustrating they don’t even know what to be angry about. Like instead of flying to Sacramento and ripping out 5000 servers why not flip out that the code has 70,000 different hard coded references to a single data center instead of one.
He did not know that at the time because his poor little brain started hurting.
Yeah, that’s basically why I got out of IT. Too many managers/clients refusing to listen to warnings about what would happen when they did X, then blaming the techies when things went to shit.
Because they are the “boss”, they have 0 accountability. Worst case for them is a golden handshake, and failing upwards where the cycle starts again.
A fractal Rube Goldberg machine
Welcome to software engineering Elon.
I do not think it is a coincidence that Tesla has recently released the updated Model 3 to some decently positive reviews. I think that is in no small part to Musk being so distracted by Twitter that he hasn’t been able to fuck up things over at Tesla in a while.
At the risk of sounding like an apologist for this prat, the frustration at being told it will take months to move systems is understandable. Also, the idiot developers who hard-coded the data center location deserve to be fired. Data center floor tiles can be removed easily with a flat blade as a lever instead of using suction cups.
Obviously a coke-fuelled man-child doing it in the middle of the night is ridiculous but if you have unlimited resources you can move any number of servers in a few days, easily. In some ways it’s impressive that he was able to pull this off on a whim without a catastrophe (the DeSantis fiasco notwithstanding). It definitely should not have been suggested by a competent data center engineer that it would take months to move anything if the CEO wants it in weeks. Even though he’s an ass, I don’t blame him for being annoyed about that.
Remember he just gutted Twitter before pulling this stunt, so the estimate of a few months to move the servers might be true if the entire department that handle infra has been gutted with only skeleton crew left.
Well, yes. If nobody left has a clue then it’s going to take a little longer but you could physically move just about anything in a few weeks with the right crew, even if you had to bring them in cold. An open checkbook solves a lot of logistical problems.
The proof is more or less self evident. If this idiot and his cousin were able to pull it off without breaking anything critical, then it stands to reason that a properly managed team would have been able to do it in a more orderly way in a few days.
I get that everyone wants to paint this as completely irresponsible, but apart from the fact that it was done so haphazardly in the dead of night and gate crashing the data center security (nobody is going to refuse access to the CEO), there’s really nothing here that’s completely out of order. Locking the gear in the trucks is pretty standard for intact secure data transport. The real mistake is the infra manager sandbagging the move estimate - or not understanding how to plan and execute it.
Physically move them is one thing. Reassigning each server into the new data center network is a whole other thing. It won’t be as simple as connecting the power and network cables. From the post, the rack density is different so you’ll probably have to change the each server name to match the new rack position. Then the hostname and subnet probably changes in the new data center, so now you’ll have to map everything again (the hard coded references to Sacramento mentioned by Musk). The 100MM contract means they have a lot of servers to account for. This is the real headache of the migration and probably the reason Twitter keep having random outages for months after this stunt. They probably took shortcut and can’t bring all those servers online in time to handle traffic bursts which leads to another Musk’s shenanigans (e.g. forbidding visitors from viewing tweets unless they’re logged in to limit servers load, etc).
Edit: the more I think about this, the more my head hurt. If any infra people reading this, what are you going to do if you suddenly received truckloads of servers yanked from another data center location and told to bring them online again ASAP, while more than half of your team has been laid off? Seriously, what’s the step you’re gonna do to bring all these servers online again? Oh, and those servers probably not gracefully shut down and just have their power cable yoinked off.
Oh by no means am I suggesting it was reasonable to do this. Musk would be a fucking nightmare as an employer. As a customer probably not much better but you know what they say about a fool and his money. This fool would be a great customer as long as you had a good lawyer to write the contracts.
I do suspect that some of the details of this story are somewhat embellished though, if only for the sheer joy of it, which I’m all for. It’s a great story. I don’t believe, for instance, that they could possibly have moved 5000 racks - or even 5000 servers - as I think the story was intimating. It sounds like they filled a few semis, which would be a small fraction of the systems. Maybe this was just the last of it that was too hard to move earlier. As for the rack configs at the other end, they would need power and services and an empty space if they are just rolling the stuff in. That’s only a few weeks of lead time in a properly run facility.
If they had their reservations set up correctly they wouldn’t need to change hostnames or even addresses, just wheel in the racks, brace and connect them. Ideally stuff would be shut down gracefully, but it shouldn’t really matter if they just pulled the plug. The software should be resilient enough to restart ok. Again, no idea if they had anything thought out, probably not, given the way it was done. But I have seen a big tech co move several rows this way when they basically couldn’t be bothered figuring out how to logically migrate them. Of course they weren’t doing it with a coked up CEO at 2am on Christmas Eve, but it wasn’t as difficult as you might imagine. But yeah not 5000 racks at once. Not even close to possible.
OMG OK that’s it. Tesla cars are now out of the question for me and if I ever get the chance to ride on a SpaceX ship (not very likely) I think I’d decline. Totally different companies ofc but the same master “mind” behind.
This guy represents everything that you do not want to see in a CEO.
NO THANK YOU
Stockton Rush was an orphan in comparison
It really seems like SpaceX worked for real. They now have the best safety record for any booster and most of the world’s space traffic. What’s their reusability record now, is it 16 flights on one rocket? You can’t argue with that result.
I don’t know what he did to get to the point of “fail fast” during development but they put their money where their mouth is. Multiple catastrophic test failures that would have been career ending anywhere else, seemingly weren’t, and they appeared to have a very fast (for rocketry) and very successful program
SpaceX is now old space. They are the launch capacity of the USA. It’s ridiculous actually
From what I understand, SpaceX made real effort to split the company into two operations. One uses the reliable Falcon 9 system launching from Cape Canaveral (and other established launch facilities) to put satellites and astronauts into space. The other operation is Elon Musk’s playground in Boca Chica where he tries to build the biggest spaceship ever!
Don’t get me wrong, there are some good engineers working at the Boca Chica operation, I’ve heard the Raptor engine is really good and there’s probably some other things they’ve made there that will be useful for rocketry in general. And who knows, the really smart people may get the biggest rocket ever to actually work someday despite Musk’s stupidity.
Batshit crazy for a normal person, just another day for Elon.
This needs to be brought up everytime someone claims Elon is purposely sabotaging Twitter. Someone actively sabotaging would not get their hands dirty going in and doing this themselves. These are the actions of the physical embodiment of Dunning-Kruger.
Elon Musk is a privileged manchild who never grew out of his teenager phase, throwing around his inherited wealth like the kid from Blank Check and throwing temper tantrums anytime someone calls him out on his bullshit. Any claims to success he may have had been entirely in spite of him, not because of him. He doesn’t have any fucking idea what he’s doing and if any one of you or I failed even a fraction as much as Musk had, we’d have all been fired ten times over.
Sounds a lot like another conservative figurehead I could name…
Actually, it sounds a lot like almost all conservative figureheads in US politics.
They’re completely idiotic and yet way more successful than I’ll ever be. But then again I have a guilty conscience.
They’re “successful” if you define success as being born into wealth and/or being willing to ignore the suffering of others. For you and me (and most right thinking people), that’s no real measure of success. Unfortunately, capitalism and the US political systems gives power to the people least suited to wielding it.
And that’s why their base votes for them. Because they can relate.
That is an insult to teenagers
Well that was a crazy read! I really feel for the workers who had to witness all that nonsense.
Imagine one day, he walks into spacex, board on a rocket, and shoot to mars… just like this!
Maybe someone should trigger him in twitter
“Elon musk is a drooling socialist cuck who is apparently far too cowardly to meet my challenge of a firm July 4th, 2024 launch date for his personal mission to mars. Jeff bezos could easily do this if he wasn’t too busy actually running the company that he owns.”
Maybe we could get him interested in submersibles…
Yeah, get him to try to debunk the English cave diver for real this time.
“Elon Musk couldn’t possibly make a submarine better than OceanGate!” 🤭
deleted by creator
So, what was he actually there to grab, while doing this little song and dance. What was on that server that he was trying to hide? 🤔🤔🤔
I don’t even think he’s that smart.
Everything in this article leads me to the same conclusion.
That reads like something out of The Hitchhiker’s Guide to the Galaxy.
Zaphod Beeblebrox, because of “an accident with a contraceptive and a time machine”, father, grandfather, and great-grandfather are actually his direct descendants.
Musk must have been the offspring of an unspeakable accident between Zaphod and one of those Sacramento racks.
That reads like something out of The Hitchhiker’s Guide to the Galaxy.
I was thinking an episode of Silicon Valley but yeah, Hitchhiker’s Guide fits too…
What a fucking moron. This is the guy whose self-driving car you trust?
Luckily, he’s not the one making them!
Nope, just the CEO of the company that has regularly overridden the decisions of his engineers, driving anyone with any actual experience away.
Autonomous vehicles built by hundreds of first-year software engineering graduates seems like a good idea though.
But but, it makes fart sounds!! /s
…what?
Teslas have an “Emissions Testing Mode” on the centre console that’s just a fart sound generator.
Nope. Once upon a time long ago I thought I wanted a Tesla. Now? I’d rather drive a Ford Focus.
I feel personally attacked here.
Well, I drive a Kia Soul, so I’m probably not one to be throwing stones.
I’d rather drive a Ford pinto.
Ah man, I love the Ford [Dick in Portuguese]
I’d rather drive an old lada made out of depleted uranium than a Tesla.
I’d rather ride a bike
For real. I love those ugly things.
The dude is a war criminal
Why is your user tagged as
banned?I posted a funny drawing that had a dick in it and got banned for posting “porn”.
You mean how he undermined US and Ukraine intelligence and leaked information to Russia? Those war crimes?
No how he shutoff Starlink to Ukraine as they were mounting an offensive to reclaim their stolen land.
Sounds treasonous
Every great genius inventor and businessman can be a little eccentric. Remember that before you decide to call them mean names like “abuser”, “megalomaniac” etc. It’s actually quirky and endearing once you factor in that he’s a genius inventor and businessman (I’m also one of the misunderstood people in the same category).
