Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "deployment story"
-
Oh, man, I just realized I haven't ranted one of my best stories on here!
So, here goes!
A few years back the company I work for was contacted by an older client regarding a new project.
The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.
Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.
Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.
Long story short, a lot bigger than what we were expecting based on the initial brief.
The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.
They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.
Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.
So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.
So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.
We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.
As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.
So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...
Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.
Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!
[Continues in the comments]26 -
Worst thing you've seen another dev do? Long one, but has a happy ending.
Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.
The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.
Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.
At 5:00PM on a Friday, deploys, goes home.
Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)
After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)
We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.
Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.
Well, since I was responsible for logging, I was next in the hot seat.
For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."
Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.
A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.
Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).10 -
Worst dev team failure I've experienced?
One of several.
Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.
I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.
The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.
3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.
3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.
4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).
About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.
3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.
About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.
Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.
A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.
Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."11 -
My code review nightmare part 3
Performed a review on/against a workplace 'nemesis'. I didn't follow the department standards document (cause I could care less about spacing, sorted usings, etc) and identified over 80 bugs, logic errors, n+1 patterns, memory leaks (yes, even in .net devs can cause em'), and general bad behavior (ex.'eating' exceptions that should be handled or at least logged)
Because 'Jeff' was considered a golden child (that's another long TL;DR), his boss and others took a major offense and demanded I justify my review, item by item.
About 2 hours into the meeting, our department mgr realized embarrassing Jeff any further wasn't doing anyone any good and decided to take matters into his own hands. Thinking 'well, its about time he did his job', I go back to my desk. About an hour later..
Mgr: "I need you in the conference room, RIGHT NOW!"
<oh crap>
Mgr: "I spoke to Jeff and I think I know what the problem is. Did you ever train him on any of the problems you identified in the review?"
Me: "Um, no. Why would I?"
Mgr: "Ha!..I was right. So lets agree the problems are partially your fault, OK?"
Me: "Finding the bugs in his code is somehow my fault?"
Mgr: "Yes! For example, the n+1 problem in using the WCF service, you never trained him on how to use the service. You wrote the service, correct?"
Me: "Yes, but it's not my job to teach him how to write C#. I documented the process and have examples in the document to avoid n+1. All he had to do was copy/paste."
Mgr: "But you never sat with Jeff and talked to him like a human being? You sit over there in your silo and are oblivious to the problems you cause. This ends today!"
Me: "What the...I have no idea what you are talking about. What in the world did Jeff tell you?"
Mgr: "He told me enough and I'm putting an end to it. I want a compressive training class developed on how to use your service. I'll give you a month to get your act together and properly train these developers."
3 days later, I submit the power-point presentation and accompanying docs. It was only one WCF with a handful of methods. Mgr approved the training, etc..etc. execute the 'training', and Jeff submits a code review a couple of weeks later. From over 80 issues to around 50. The poop hits the fan again.
Mgr: "What's your problem? When are you going to take your responsibility seriously?"
Me: "Its pretty clear I don't have the problem. All the review items were also verified by other devs. Its not me trying to be an asshole."
Mgr: "Enough with the excuses. If you think you can do a better job *you* make the code changes and submit them for Jeff for review. No More Excuses!"
Couple of days later, I make the changes, submit them for review, and Jeff really couldn't say too much other than "I don't see this as an improvement"
TL;DR, I had been tracking the errors generated by the site due to the bugs prior to my changes. After deployment, # of errors went from thousands per hour to maybe hundreds per day (that's another story) and the site saw significant performance increases, fewer customer complaints, etc..etc.
At a company event, the department VP hands out special recognition awards:
VP: "This award is especially well earned. Not only does this individual exemplify the company's focus on teamwork, he also went above and beyond the call of duty to serve our customers. Jeff, come on up and get this well deserved award."19 -
The world makes no fucking sense.
In 2013 I had a manager approve a couple days' leave coz my son was having medical issues.
He was super nice about it and told me I could take as much time as I needed. I said, a couple days is enough. I took Thursday and Friday off. I took two days.
On Monday, an emergency meeting was held with the CTO (it was a small company, it went me -> manager -> C suite). I was told that a production deployment happened on Friday that fucked up a few clients' systems and that it had cost said clients hundreds of thousands dollars and are now suing the company.
Turns out on Friday, lead developer was also given the day off for whatever reason and I was being scolded because as the next senior developer, it was my responsibility to review code and make sure shit like this doesn't happen.
I agreed (and still agree) but also explained I had already filed leave weeks prior and I wasn't informed about dev lead's absence. Sure I could've checked my messages but my kid was in the hospital and I was busy. Still I couldn't help but feel a little guilty.
Manager holds a separate meeting with me and talks me into just writing an apology note in the email chain and he'll do the rest of the talking for me and make sure I get minimal punishment. I trusted him, he was the one who found me and brought me into the company (I know, I was naive).
So I wrote the email. It was a small note. I apologized for not checking messages and explained my situation again and mentioned I would've definitely checked if I was informed that the lead dev would be away.
Another meeting was held the next day and after pleasantries the Manager started with this, "Ok so we've all seen the email and understand that this was all Angry's fault right?".
Now, we're not native English speakers and Manager doesn't really do well with grammar. I was alarmed by what he said but wasn't angry because I was pretty sure that's not what he meant. I'm sure he meant to say that "Angry feel's guilty but his actions were understandable given the circumstance" or that he forgot a "not" in there and really meant "not Angry's fault". Surely this is what he meant to say. Right?
But then the rest of the meeting went on and I was unceremoniously let go. Immediately for "failing to accomplish my tasks and costing the client 100Ks of dollars". I wasn't even given a chance to say anything else.
The meeting ended and since we were both in the office, Manager approached me with exit papers and a check (~1200 USD)--it was my month's pay. I was asked to leave that day and was told I didn't need to come back. No handovers, no knowledge transfers, not a even a documentation of open projects I was handling.
I realized I just was made the scapegoat by a management screwup that costed our clients a lot of money.
Of course, I wrote the CEO multiple emails the next couple days. I also cc'd the CTO. No response.
A couple of weeks pass, I get another job at a cool company and i promptly move on.
I write this story now because I just found out today that in 2016, Manager was let go by the company for **sexual harassment**. Apparently, he actually did it too according to friends I still had within the company.
Here's where it gets fucked up. He turns and sues the company for unlawful termination and I guess to avoid a long legal battle? the company settled. They fucking settled and handed this man 2 Million PHP (at the time about 40k USD).
2 fucking million. Life changing money around here. And he got it by being a slimy piece of shit.
The world makes no fucking sense.10 -
TL;DR I'm fucking sick and tired of Devs cutting corners on security! Things can't be simply hidden a bit; security needs to be integral to your entire process and solution. Please learn from my story and be one of the good guys!
As I mentioned before my company used plain text passwords in a legacy app (was not allowed to fix it) and that we finally moved away from it. A big win! However not the end of our issues.
Those Idiot still use hardcoded passwords in code. A practice that almost resulted in a leak of the DB admin password when we had to publish a repo for deployment purposes. Luckily I didn't search and there is something like BFG repo cleaner.
I have tried to remedy this by providing a nice library to handle all kinds of config (easy config injection) and a default json file that is always ignored by git. Although this helped a lot they still remain idiots.
The first project in another language and boom hardcoded password. Dev said I'll just remove before going live. First of all I don't believe him. Second of all I asked from history? "No a commit will be good enough..."
Last week we had to fix a leak of copyrighted contend.
How did this happen you ask? Well the secure upload field was not used because they thought that the normal one was good enough. "It's fine as long the URL to the file is not published. Besides now we can also use it to upload files that need to be published here"
This is so fucking stupid on so many levels. NEVER MIX SECURE AND INSECURE CONTENT it is confusing and hard to maintain. Hiding behind a URL that thousands of people have access to is also not going to work. We have the proof now...
Will they learn? Maybe for a short while but I remain sceptic. I hope a few DevrRanters do!7 -
this is how I destroyed my career in IT and how I'm headed to a bleak future.
I've spent the last 10 years working at a small company developing a web platform. I was the first developer, I covered many roles.
I worked like crazy, often overtime. I hired junior dev, people left and came. We were a small team.
I was able to keep the boat afloat for many years, solving all the technical problems we had. I was adding value to the company, sure, but not to mine professional career.
There was a lot of pressure from young developers, from CEO, from investors. Latent disagreement between the COO and the CEO. I was in between.
Somehow, the trust I built in 10 years, helping people and working hard, was lost.
There was a merge, development was outsourced, the small team I hired was kept for maintenance and I was fired, without obvious explanations.Well, I was the oldest and the most expensive.
Now I'm 53, almost one year unemployed.
I'm a developer at heart, but obsolete. The thing we were doing,
were very naif. I tried to introduce many modern and more sophisticated software concepts. But basically it was still pure java with some jquery. No framework. No persistency layer, no api, no frontend framework. It just worked.
I moved everything to AWS in attempt to use more modern stack, and improving our deployment workflow.
Yes, but I'm no devop. While I know about CD/CI, I didn't set up one.
I know a lot of architectural concepts, but I'm not a solution architect.
I tried to explain to the team agile. But I'm not a scrum master.
I introduced backlog management, story mapping, etc. But I'm not a product manager.
And before that? I led a team once, for one year, part of a bigger project. I can create roadmap, presentations, planning, reports.
But I'm not a project manager.
I worked a lot freelancing.
Now I'll be useless at freelancing. Yes I understand Angular, react, Spring etc, I'm studying a lot. But 0 years of experience.
As a developer, I'm basically a junior developer.
I can't easily "downgrade" my career. I wish. I'll take a smaller salary. I'll be happy as junior dev, I've a lot to learn.
But they'll think I'm overqualified, that I'll leave, so they won't hire me even for senior dev. Or that I won't fit in a 25 y.o. team.
My leadership is more by "example", servant leader or something like that. I build trust when I work with somebody, not during a job interview.
On top of that, due to having worked in many foreign countries, and freelancing, my "pension plan" I won't be able to collect anything. I've just some money saved for one year or so.
I'm 53, unemployed. In few years time, if I don't find anything, it will be even harder to be employed.
I think I'm fucked25 -
I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.
I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.
We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.
The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.
We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.
It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.
Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.
But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.
I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.
But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.
Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.
They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.
I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this
Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.
I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.
But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.
... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.rant code application bad project management annoying as hell bad code c++ bad client bad design application development16 -
devRant is awesome, but Disney also manages to light-up my day.
This is how Wall-E became a beloved member of our team, and helped me put a smile on my face throughout a very frustrating project.
It all started in a company, not so far far away from here, where management decided to open up development to a wider audience in the organization. Instead of continuing the good-old ping-pong between Business and IT...
'not meeting my expectations' - 'not stated in project requirements'
'stuff's not working - 'business is constantly misusing'
'why are they so difficult' - 'why don't they know what they really want'
'Ping, pong, plok... (business loses point) ping, pong'
... the company aimed to increase collaboration between the 2 worlds, and make development more agile.
The close collaboration on development projects is a journey of falling and getting back up again. Which can be energy draining, but to be honest there is also a lot of positive exposure to our team now.
The relevant part for this story is that de incentive of business teams throughout these projects was mainly to deliver 'something' that 'worked'. Where our team was also very keen on delivering functionality that is stable, scalable, properly documented etc. etc.
We managed to get the fundamentals in place, but because the whole idea was to be more agile or less strict throughout the process, we could not safeguard all best-practices were adhered to during each phase of a project. The ratio Business/IT was simply out of balance to control everything, and the whole idea was to go for a shorter development lifecycle.
One thing for sure, we went a lot faster from design through development to deployment, high-fives followed and everybody was happy (for some time).
Well almost everybody, because we knew our responsibility would not end after the collection of credits at deployment, but that an ongoing cycle of maintenance would follow. As expected, after the celebrations also complaints, new requirements and support requests on bug fixes were incoming.
Not too enthusiastic about constantly patching these projects, I proposed to halt new development and to initiate a proper cleaning of all these projects. With the image in mind of a small enthusiastic fellow, dedicated to clean a garbage-strewn wasteland for humanity, I deemed "Wall-E" a very suited project name. With Wall-E on board, focus for the next period was on completely restructuring these projects to make sure all could be properly maintained for the future.
I knew I was in for some support, so I fetched some cool wall papers to kick-start each day with a fresh set of Wall-E's on my monitors. Subsequently I created a Project Wall-E status report, included Wall-E in team-meetings and before I knew it Wall-E was the most frequently mentioned member of the team. I could not stop to chuckle when mails started to fly on whether "Wall-E completed project A" or if we could discuss "Wall-E's status next report-out". I am really happy we put in the effort with the whole team to properly deploy all functionality. Not only the project became a success, also the idea of associating frustrating activities with a beloved digital buddy landed well in our company. A colleagues already kickstarted 'project Doraemon', which is triggering a lot of fun content. Hope it may give you some inspiration, or at least motivate you to watch Wall-E!
PS: I have been enjoying the posts, valuable learnings and fun experiences for some time now. Decided to also share a bit from my side, here goes my first rant!3 -
The story of the shittiest, FUCKING WORST day of work.
TLDR: shitty day at work, car crash to end the day.
So, let tell you about what could possibly be the worst day I had since I started working.
This morning, my alarm didn't work, woke up 30 minutes before an appointment I had with a client.
Arrived late at the client, as I start deploying. They don't have any way to transfer the deployment package to the secured server. Lost 45 minutes there.
Deployment goes pretty well. My client asks me to stay while they load some data into the app. Everything's pretty easy to work out. Just need to input 3 CSV with the correct format (which the client defined since the beginning).
I end up watching an Excel Macro called "Brigitte" (I'm not fucking kinding, could'nt have thought of that) work for 4 hours straight. Files are badly formatted and don't work.
Troubbleshooting thoses files with a fucking loader that does not tell you anything about why it failed (our fault on that one)
I leave the client at 7:30pm, going back at work, leave at 9pm.
At this point, I just want to buy some food, go home and watch series.
But NO, A FUCKING MORRON OF A BUS DRIVER had to switch lanes as I was overtaking him. Getting me crushed between the bus and the concrete blocks.
Cops were fucking dickheads, being very mean even tho I was still shaking from the adrenaline.
In conclusion, the day could have been worst. The devs at the clients are pretty cool guys and we actually had some fun troubleshooting. At work, there was still one of my colleagues who cheered me up telling me about his day.
And when I think of it, I could have got really hurt (or even worst) in the crash.
A bad day is a bad day, tomorrow morning I'm still going to get up and go to a job I love, with people I love working with.
Very big rant (sorry about that if someone's still reading)9 -
Don't waste your time - they said.
Use Spring - the good ol' framework - they said
It's not slow - they said
me: ignores them, builds a custom jetty-based webserver with the same functionality Spring+tomcat can offer (mappings, routings, etc).
My app: boots up completely in <300ms, while Spring tutorials say a hello-world app takes 3+ seconds to spin-up http://websystique.com/spring-boot/...
me: already set for deployment in lambda. I bet I can tune it up even further with lazy-loading if I really have to...
Moral of the story: sometimes bare-bones solution is a better choice: more performant, more extendable, more testable, more lightweight.
That, dear folks, is the classic LESS IS MORE :)12 -
Soo, my manager asked me to create tool for CSI. Sort of ticketing tool for service improvements.
So I spent a few months working on it including design, websocket based real time statistics, exports to their belowed excel, easy to use, fast and so on.
I've presented it to mgmt, told them that deployment was easy and just need a simple linux virtual and all is automated.
They told me that they don't have a server. Company where main business is cloud services. Didn't pay me a penny for my effort even though worked on that mostly in my free time.
I didn't even want anything for the tool, just for my time.
Then a month later they've introduced similar thing based on Sharepoint with 1/10th of fuctionality, slow as hell, buggy, unintuitive.
And guess what, I can't open source my tool because it is a company property.
So, fuck it, never gonna do anything again without proper contract, even if for the same department.
I've already left that hell hole, but thought I would share my story. -
Worked on project that has multiple forms. After deployment colleague asks me if I made the Chrome in-build autofill form function. He really liked and loved the option. Colleague calls himself front-end developer. True story!1
-
I got assigned to work on a new project a couple of weeks ago. We got the POC code handed off from senior management, since he came up with the idea over the weekend. The project concept is hella exciting, but the dev manager and PO I have to deal with make life unbearable to say the least.
We have only 2 devs (including me) and 1 QA on this supposedly very important project. Of course, management announced the project to the clients already, so now we have to deliver ASAP cause it adds “sizzle”.
The MVP deadline is... no one knows when, either July 30th or September 1st. The MVP requirements are... unknown. I swear if someone saw the list of tasks and issues attached to “MVP” Epic, they would call us nuts trying to fit it all in.
To make things better, each PR requires 2 reviewers, so we end up adding manager as a reviewer just cause we need him to hit that “approve” button. So in attempt to make life easier, we requested to have a third developer. We are getting another developer, but that guy doesn’t know how to unit test a pure function...
Current priorities are... unit testing with coverage of 95% and if we want to refactor code, we have to add area to the list in a Google Doc. As a result, we are not tackling big things like risk of SQL injections not to mention big features like i18n (5-6 languages to support by the way and yes, it’s part of MVP as well as SSR no one knows why). Currently, I spend 2-3 hours a week in calls with the team just to figure out what the hell MVP is, what we have to do and why we have to do it. Last time we spent an hour refining 1 spike and breaking down one story into 3.
Oh, we also don’t have a deployment plan, not even to test environments since DevOps team was not aware of this project at all. Thus, QA cannot create any test suites and have to test everything manually which eats a lot of their time.
This whole project is a big hot mess and I’m considering leaving it all together especially since I’m working on two squads at the same time. I love the project, I love the idea, but management makes it unbearable, so I’m not even motivated to work on that.3 -
Rookie me,back when I was making my first Android app...
After a lot of hours put into it, finally finished it...pretty much was ready for deployment...some final touches...but oh wait... I was messing with some files-wanted to delete something and misclicked the whole folder...well my brain farted for a moment and clicked yes-I managed to erase the whole fucking app I was working for months...!
The whole world shut down at that moment! What the fuck did I do?...
In this point I want to thank jetbrains for their magical revert button....
Moral of the story: Learn Git, backup everything and don't be too excited and fuck up tremendously...! -
When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.
Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.
When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.
Speaking of which, there's more than six microservices per developer!
Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?
When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:
1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.
Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).
Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.
When confronted/advised about these issues the response from the CTO/CEO
varies:
(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".
Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.
There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".
It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?2 -
Never got scheduled for night deployment shift. Once i did, everything suddenly catches fire.
Well, twice. In a row.1 -
I started working for a startup as Server Administrator/ System Integrator beside university to get some dollars with easy work and nice people.
((I Know two of the C*Os so I got a had feeling with this. Besides the upcoming story I'm still really happy with my position and career chances here. God bless my Department which has the most funny/rude guys, love you.))
tl;dr:
Guy fakes his Skillset and fuckup whole department, can´t do most of his basic tasks. I had my first and hopefully last interaction with this bastard.
Heres how everything started:
I was more and more involved in the leading processes and decisions.
Heard about a story where and why the whole dev-department was kicked out of his position because they were crappy developers. And cant just believe the stories they told me about the former Dev-Lead
Now I met the former "Development Lead"
I was brought in because we in the IT wondered why he would like to share his local machine password with colleges. After some questions he came out with the Reason.
He is doing home-office for some days a week now and wants his colleges to be able to start his "software". (already confused by that)
The "better IT-guy" in me offered help for automatic deployment CI/CD stuff so that they can use it as an inhouse service.
BIG OOF incoming:
"The code is not in git because I wanted to clean it up before"
"My IDE is the only place where my PHP crap work is running"
"The 'PHP-software' is to complex for this"
My Lead and I were completely speechless,
I understand the decision to kick this "dev-Lead" from the lead position down to a code monkey/ script kid.
Now I´m thinking about getting my Hands on the Lead position after my exams because if such bastards with no clue about basic stuff, no clue about leading, no clue about ci/cd, no clue about generic software stuff get the job I would easily be the "good IT-guy" with more responsibility/ skill.
Now I sit here, hate people that fake their skills and set back work of colleges for multiple months and never asked for help or advice.
And the little "Bastard Operator from Hell" in my just wants to delete all his files, emails account during a migration to completely demotivate the person who failed to be responsible for a team nor their projects.rant ci/cd php administrator startup script-kid i hate people unskilled skill faker lead developer devops5 -
Fuuuuuuuck!!
CR estimates:
Part 1: 2h including testing
Part 2: 2h-2days-maybe never (small changes on horrifically fucked up project noone understands with tons of tech debt)
Managed to pull off the part two in one day.. //yay me?!
Additional day to unfuckup git fuckups (including but not limited to master head not compiling because a smartass included *.cs in .gitignore file which he also pushed..don't ask, I have no clue why..) which was a huuuge deal for me as I usually use only local repo and had no idea how to tackle this.. coworker helped out.. seems I was on the right way, but git push branchy was acting up & said I had to login & ofc I had no clue what the pass was set to (first setup was more than 2yrs ago)..so new key, new pass.. all good.. yay!
Back to the original story/rant: Now I'm stuck with writing jira explanation why it was done this way & not the way customer suggested. They offered only vague description anyways which would require me to do a hacky messy thing, ew.. + it most probably would require major data modifications after deployment to even make it work..
Anyhow, this expanation is also easy peasy in english..
BUT...
I must write it in my native tongue.. o.O FML! Spent almost 40mins on one paragraph..
Sooo.. if anyone will petition to ban non english in IT, I'm all for it!!2 -
!rant, but story
Update: I now work as a 1st lvl support
After 5 to 11 months I'll work in deployment and after that in development.14