This was recent, and it’s still pretty sore for me. I doubt anyone will be able to pinpoint who I am, but if you for some reason are in this forum and recognize me please DM me. Try to count all of the red flags.
I was hired as a software engineer and was immediately thrown onto a “high-visibility project”. My service was the middle man between two other mission critical services. Essentially downstream provided metrics and needed to get to upstream.
I laid out several different architectures that I recommended. First was prometheus. It’s literally designed to do this, downstream is spread across many servers, prometheus is literally built to do this. Upstream then can scrape prometheus, any other future dependents can also scrape. This was rejected. “We did prometheus once, it didn’t work.” I check, it’s a single tenant instance of Prometheus running on one 24XXL AWS VM. So, they didn’t know how to properly configure prometheus. I tell them I can kill 2 birds with one stone. No, prometheus bad. Rejected.
Second, we use a highly reliable queue setup. Downstream publishes to queue, Upstream reads from queue. Seems simple enough, can have many producers and many subscribers, and we already have a kafka service. Rejected. For why, I ask. Literally “Upstream doesn’t know how to work with queues”. Literally got that as an answer. Read that as “We need to choose a subpar architecture because we openly admit our engineers don’t have the necessary skills”. I even offered to help them, to write that part of the code. Rejected.
Third option came straight from the CTO. We love datadog here. Everyone does. Datadog. Oh you feel that pit in your stomach don’t you. The mandate came down from on high that Downstream would push metrics to Datadog. I then would need to periodically scrape Datadog, and then have an API that the upstream could then periodically scrape me. I looked into Datadog’s API. They don’t really support this. I reach out to Datadog, talk to their engineers, and they confirm this is a horrible pattern. I bring this up, say it’s just not a good decision, there are better ways. Literally rejected by the CTO himself.
So, I build this rickity ass service, brand new built with thumbtacks and glue. Along the way more is mandated to me. We’ll have literally 8x the number of metrics we originally planned for. We’re well over Datadog’s API limitations. I am mandated to put it into a Postgres instance. Every decision I am overridden.
On top of this, Downstream is completely overworked and doesn’t have time to answer questions about specific metrics. Upstream then asks me, who has been there now for a grand total of 4 months, and I don’t know the specific questions. I refer them to Downstream for helping describe what specific metrics are and do. They report to my superiors that I am not being a team player for this. They also don’t know how to use my API, I have to explain concepts like GET and POST to them, how to serialize datetimes. I end up writing some of their code for them just to make it work.
In the end, we shipped late. There was an arbitrary deadline set by the CTO that we missed - we were not consulted on this deadline, there was no reason for the deadline beyond “We should be live on this date”. We missed it by 5 days. During those 5 days I am online every waking moment, sleeping an average of 4-5 hours per night. I’m a walking zombie trying to patch this thing.
A week after release I’m called onto a meeting with my direct boss, who reports to the CTO. He tells me that due to my “Lack of Ownership” and “Lacking team spirit” they are letting me go. I’m stunned. This entire time literally any decision I tried to make was overridden. They chose the worst possible architecture, forced me to implement it, forced me to talk to third parties about designing this anti-pattern. I had 2 other teams actively work against me, and on top of that I had no support from anyone. I was alone, and isolated. I got off that call, and I just cried. I felt like such a failure.
I’m at a new job now, and I’ve realized what a toxic environment that was. Horrible engineering practices, way too much pressure on me alone. I had developed health issues that I wasn’t even aware of that now have subsided. I literally tried my best, and they just let me go. I found out later that my boss who fired me was being chewed out over the horrid project, and he put 100% of the blame on me to save his own ass.
So, they didn’t know how to properly configure prometheus. I tell them I can kill 2 birds with one stone. No, prometheus bad. Rejected.
Bruh I would trust a high schooler to successfully spin up prometheus; stories like this are what make more grateful for my job, brings back memories of insane posts on r/sysadmin
They had one single Prometheus node for the entire company and couldn’t figure out why it needed so much CPU. They never once thought that maybe they set it up incorrectly
Oh man, i felt that one. Thats full on red flags and if i were in your shoes, and being fired didnt stun me, id have sent a mail to all teams and managers involved (including ceo and cto) and shown the emails and reports showing the toxicity. Wouldnt have gone down without a final fuck you haha.
That said, sometimes its best to let it go and start something new somewhere else, like you did.
Im also in a semi similar scenario at the moment. They want me, and 2 others, to make a new version of their program in .net. Thats all fine and dandy. We get functional input when asked and all that.
We chose to create .net code/windows that are executed/called from inside the old client. However, when push comes to shove there is no support whatsoever. The other client team doesnt want to implement our stuff and delays it as much as they can. The server /api team does whatever the fuck they want, constantly breaking everything and choosing anti-patterns on their end. Manager will (and suspect he has already) thrown me and the architect under the bus because we have exposed the bullshit of said server team.
Besides functional input we are completely on our own with no support, on the contrary. We have to fight for every choice or design. And god knows when our code will get released… Its been 3 fucking years and our code has been done for a very long time.
Hell, even release was hard. “Can i push release from devops pipelines?” “No.”
“Can i execute the needed script automatically from devops?” “No”
“You want me to do release manually?” “Yes”. " not going to happen mate".
I also made several packages and helpful tools to make communication with the system easier, but dont you dare think anyone has promoted that internally to other teams. Hell, when i do get feedback and report that, nothing gets done with it. No ask or preasure to server team to implement the api calls.
So many issues they could solve if they just fucking listen
Id send the mail i was talking about here too when i get out, but im a consultant. I should not throw my company under the bus for this when we find a new project for me to do but ive been really badly wanted to do it and have the ceo in cc because fuck that
People like you and i should not doubt ourselves because of shit like this. From your story you know your stuff and listen to what the team has to say, you consider their experiences and what they are prepared to do. Thats good skills for an architect!
Know that I completely empathize with everything here, even the consultancy bits. It kills me when companies won’t do basic things like CI/CD. In fact, that’s one of the major red flags. If a company isn’t minimum doing some sort of pipeline to go to prod it’s pretty much a no from me at this point. It’s such an important critical security step that if you didn’t do that, you’re not doing a lot else either.
Sounds like you’re where I was, but you’ve had a longer timeline luckily. Don’t let them blame you, don’t let them put it on you. Sounds like you’re walking that political tightrope pretty well.
Thank you for the kind words, I’ve been feeling pretty low since then and your last few words made me smile :)
Jesus fucking christ, that’s incredible. I don’t know what any of that software engineering stuff is, but based on your description of events, it just sounds like a waking nightmare.
If it makes you feel better, companies like that where upper management are quick to throw people under the bus for their own fuck ups are usually not long for this world. They’ll continue to make bad decisions and the effects will be felt both downstream with low employee morale and retention rates, as well as commercial loss for the company. You are much better off being valued for your work elsewhere.
Thanks, basically take it as you do something well, and they hire you because you do it well. They ask you to do the thing you do well, you build out plans, you build out solid options, write it all up and present it all. Then your manager who has about 4 months of experience doing what you do for a living tells you that he doesn’t like it, and that you should do it in the worst possible way forward. You don’t get a say. Then it all rolls from there.
It does make me feel better. I do know that morale tanked after I left. Not me being self centered, I’ve had multiple people tell me that there was a noticeable drop in morale because people knew it was bullshit. They had to have several meetings saying “They didn’t fit our culture” but my colleagues weren’t stupid. They knew who I was and what I was like, they knew it was a political move. I don’t expect they’ll all leave because of me or anything, but I do hope that it’s just the first couple rocks in an avalanche.
I did a lot of consulting. Companies like that include well-respected, well-known firms. They are so profitable that they can afford to have utter shit management in non-core functions.
A nightmare. I’ve never had anything quite that bad, but I’ve had plenty of work experience where management is making bad decisions and has no accountability.
My current role is hourly, so I’m happy to shut the laptop exactly 8 hours into the day. Pays a lot less, sadly.
That 8 hours is a godsend. You reminded me of another story there. My first week, my first few days I went into office. I had about 5 hours of work because you know, onboarding. After that I sat around for an hour or two, asking people what to do, reading documentation, you know. After 7 hours I was like (to the same manager), do you have anything else for me? Or should I get going for day. He said nope! Great first day, see you tomorrow! I literally thought nothing of it, it’s the first week, that’s how all first weeks are. Onboarding is always slow.
That bastard sat me down in my second week saying they had a strict 8 hour in office policy and that “I had been noticed”. I reminded him that he was the one who said I could leave early and that on even that day I was in earlier than he was. I learned that it didn’t matter if I showed up at 7am, if I left at 3 they would think I’m “leaving early”. My ride unfortunately dropped me off every day at 7, so I ended up having to work 9 hour days every day in office so they wouldn’t think I was slacking off.
That sucks. It sounds like a dodged bullet and I wouldn’t take it personally.
I’ve seen this happen from different angles where a manager or c-suite has them hanging people out to dry so they can protect their own ego. Even if you managed to get through that situation, you’ll feel like you’re walking on glass for every project and that’s just not a way you want to live.
That’s exactly how it is. Every project now I’m worried everyone is against me and that one slip up will be it. My confidence is at zero and all because that of that shit manager. I appreciate the words, it’s something I’ll have to overcome over time.
This was recent, and it’s still pretty sore for me. I doubt anyone will be able to pinpoint who I am, but if you for some reason are in this forum and recognize me please DM me. Try to count all of the red flags.
I was hired as a software engineer and was immediately thrown onto a “high-visibility project”. My service was the middle man between two other mission critical services. Essentially downstream provided metrics and needed to get to upstream.
I laid out several different architectures that I recommended. First was prometheus. It’s literally designed to do this, downstream is spread across many servers, prometheus is literally built to do this. Upstream then can scrape prometheus, any other future dependents can also scrape. This was rejected. “We did prometheus once, it didn’t work.” I check, it’s a single tenant instance of Prometheus running on one 24XXL AWS VM. So, they didn’t know how to properly configure prometheus. I tell them I can kill 2 birds with one stone. No, prometheus bad. Rejected.
Second, we use a highly reliable queue setup. Downstream publishes to queue, Upstream reads from queue. Seems simple enough, can have many producers and many subscribers, and we already have a kafka service. Rejected. For why, I ask. Literally “Upstream doesn’t know how to work with queues”. Literally got that as an answer. Read that as “We need to choose a subpar architecture because we openly admit our engineers don’t have the necessary skills”. I even offered to help them, to write that part of the code. Rejected.
Third option came straight from the CTO. We love datadog here. Everyone does. Datadog. Oh you feel that pit in your stomach don’t you. The mandate came down from on high that Downstream would push metrics to Datadog. I then would need to periodically scrape Datadog, and then have an API that the upstream could then periodically scrape me. I looked into Datadog’s API. They don’t really support this. I reach out to Datadog, talk to their engineers, and they confirm this is a horrible pattern. I bring this up, say it’s just not a good decision, there are better ways. Literally rejected by the CTO himself.
So, I build this rickity ass service, brand new built with thumbtacks and glue. Along the way more is mandated to me. We’ll have literally 8x the number of metrics we originally planned for. We’re well over Datadog’s API limitations. I am mandated to put it into a Postgres instance. Every decision I am overridden.
On top of this, Downstream is completely overworked and doesn’t have time to answer questions about specific metrics. Upstream then asks me, who has been there now for a grand total of 4 months, and I don’t know the specific questions. I refer them to Downstream for helping describe what specific metrics are and do. They report to my superiors that I am not being a team player for this. They also don’t know how to use my API, I have to explain concepts like GET and POST to them, how to serialize datetimes. I end up writing some of their code for them just to make it work.
In the end, we shipped late. There was an arbitrary deadline set by the CTO that we missed - we were not consulted on this deadline, there was no reason for the deadline beyond “We should be live on this date”. We missed it by 5 days. During those 5 days I am online every waking moment, sleeping an average of 4-5 hours per night. I’m a walking zombie trying to patch this thing.
A week after release I’m called onto a meeting with my direct boss, who reports to the CTO. He tells me that due to my “Lack of Ownership” and “Lacking team spirit” they are letting me go. I’m stunned. This entire time literally any decision I tried to make was overridden. They chose the worst possible architecture, forced me to implement it, forced me to talk to third parties about designing this anti-pattern. I had 2 other teams actively work against me, and on top of that I had no support from anyone. I was alone, and isolated. I got off that call, and I just cried. I felt like such a failure.
I’m at a new job now, and I’ve realized what a toxic environment that was. Horrible engineering practices, way too much pressure on me alone. I had developed health issues that I wasn’t even aware of that now have subsided. I literally tried my best, and they just let me go. I found out later that my boss who fired me was being chewed out over the horrid project, and he put 100% of the blame on me to save his own ass.
Thanks for listening
Love it when the CTO is a guy who’s work experience was previously selling bikinis, but surely he’ll do fine managing the technical team.
Bruh I would trust a high schooler to successfully spin up prometheus; stories like this are what make more grateful for my job, brings back memories of insane posts on r/sysadmin
They had one single Prometheus node for the entire company and couldn’t figure out why it needed so much CPU. They never once thought that maybe they set it up incorrectly
Oh man, i felt that one. Thats full on red flags and if i were in your shoes, and being fired didnt stun me, id have sent a mail to all teams and managers involved (including ceo and cto) and shown the emails and reports showing the toxicity. Wouldnt have gone down without a final fuck you haha.
That said, sometimes its best to let it go and start something new somewhere else, like you did.
Im also in a semi similar scenario at the moment. They want me, and 2 others, to make a new version of their program in .net. Thats all fine and dandy. We get functional input when asked and all that.
We chose to create .net code/windows that are executed/called from inside the old client. However, when push comes to shove there is no support whatsoever. The other client team doesnt want to implement our stuff and delays it as much as they can. The server /api team does whatever the fuck they want, constantly breaking everything and choosing anti-patterns on their end. Manager will (and suspect he has already) thrown me and the architect under the bus because we have exposed the bullshit of said server team.
Besides functional input we are completely on our own with no support, on the contrary. We have to fight for every choice or design. And god knows when our code will get released… Its been 3 fucking years and our code has been done for a very long time.
Hell, even release was hard. “Can i push release from devops pipelines?” “No.”
“Can i execute the needed script automatically from devops?” “No”
“You want me to do release manually?” “Yes”. " not going to happen mate".
I also made several packages and helpful tools to make communication with the system easier, but dont you dare think anyone has promoted that internally to other teams. Hell, when i do get feedback and report that, nothing gets done with it. No ask or preasure to server team to implement the api calls.
So many issues they could solve if they just fucking listen
Id send the mail i was talking about here too when i get out, but im a consultant. I should not throw my company under the bus for this when we find a new project for me to do but ive been really badly wanted to do it and have the ceo in cc because fuck that
People like you and i should not doubt ourselves because of shit like this. From your story you know your stuff and listen to what the team has to say, you consider their experiences and what they are prepared to do. Thats good skills for an architect!
Know that I completely empathize with everything here, even the consultancy bits. It kills me when companies won’t do basic things like CI/CD. In fact, that’s one of the major red flags. If a company isn’t minimum doing some sort of pipeline to go to prod it’s pretty much a no from me at this point. It’s such an important critical security step that if you didn’t do that, you’re not doing a lot else either.
Sounds like you’re where I was, but you’ve had a longer timeline luckily. Don’t let them blame you, don’t let them put it on you. Sounds like you’re walking that political tightrope pretty well.
Thank you for the kind words, I’ve been feeling pretty low since then and your last few words made me smile :)
Jesus fucking christ, that’s incredible. I don’t know what any of that software engineering stuff is, but based on your description of events, it just sounds like a waking nightmare.
If it makes you feel better, companies like that where upper management are quick to throw people under the bus for their own fuck ups are usually not long for this world. They’ll continue to make bad decisions and the effects will be felt both downstream with low employee morale and retention rates, as well as commercial loss for the company. You are much better off being valued for your work elsewhere.
Thanks, basically take it as you do something well, and they hire you because you do it well. They ask you to do the thing you do well, you build out plans, you build out solid options, write it all up and present it all. Then your manager who has about 4 months of experience doing what you do for a living tells you that he doesn’t like it, and that you should do it in the worst possible way forward. You don’t get a say. Then it all rolls from there.
It does make me feel better. I do know that morale tanked after I left. Not me being self centered, I’ve had multiple people tell me that there was a noticeable drop in morale because people knew it was bullshit. They had to have several meetings saying “They didn’t fit our culture” but my colleagues weren’t stupid. They knew who I was and what I was like, they knew it was a political move. I don’t expect they’ll all leave because of me or anything, but I do hope that it’s just the first couple rocks in an avalanche.
I did a lot of consulting. Companies like that include well-respected, well-known firms. They are so profitable that they can afford to have utter shit management in non-core functions.
When this job is frustrating, it can be the most frustrating of any type of job.
Sorry and thank god you got out.
A nightmare. I’ve never had anything quite that bad, but I’ve had plenty of work experience where management is making bad decisions and has no accountability.
My current role is hourly, so I’m happy to shut the laptop exactly 8 hours into the day. Pays a lot less, sadly.
That 8 hours is a godsend. You reminded me of another story there. My first week, my first few days I went into office. I had about 5 hours of work because you know, onboarding. After that I sat around for an hour or two, asking people what to do, reading documentation, you know. After 7 hours I was like (to the same manager), do you have anything else for me? Or should I get going for day. He said nope! Great first day, see you tomorrow! I literally thought nothing of it, it’s the first week, that’s how all first weeks are. Onboarding is always slow.
That bastard sat me down in my second week saying they had a strict 8 hour in office policy and that “I had been noticed”. I reminded him that he was the one who said I could leave early and that on even that day I was in earlier than he was. I learned that it didn’t matter if I showed up at 7am, if I left at 3 they would think I’m “leaving early”. My ride unfortunately dropped me off every day at 7, so I ended up having to work 9 hour days every day in office so they wouldn’t think I was slacking off.
That sucks. It sounds like a dodged bullet and I wouldn’t take it personally.
I’ve seen this happen from different angles where a manager or c-suite has them hanging people out to dry so they can protect their own ego. Even if you managed to get through that situation, you’ll feel like you’re walking on glass for every project and that’s just not a way you want to live.
That’s exactly how it is. Every project now I’m worried everyone is against me and that one slip up will be it. My confidence is at zero and all because that of that shit manager. I appreciate the words, it’s something I’ll have to overcome over time.