Opinions expressed here belong to your're mom
I have been full-time a Linux user for about 10 years now, and a professional Linux user for most of that time. I say "professional Linux user" because I feel it is more descriptive then the long list of different job titles that I've held where the responsibilities basically boil down to being really good at the commandline. One thing that I have noticed is that a lot of people who wind up in this position develop a sort of imposter syndrome. I'm not entirely excluded from this list - I have felt like I was just pretending to know what I was doing. However, I don't really feel that way anymore and I think it might be valuable to look back at a decade of experience and re-assess the things that changed my mindset from then to now.
One job that I worked had me helping on a big project to replace the webapp that ran basically the entire company. It had all the bells and whistles and did everything:
The tool in question was an off-the-shelf offering from some external company that sold it to basically everyone in the industry. It required a lot of infrastructure. Most of the servers that I worked to build and maintain at the time were primarily used to support this one tool. We had IT people in three different teams supporting different parts of this tool. When the time came to replace it because the software was reaching end-of-life, we didn't have the manpower, knowledge, or bandwidth to do it ourselves. Sensibly, we hired the company who designed and sold us the software to help us with the migration.
Given that these outsiders were going to be signing into our
production systems that handled all sorts of vital information,
management asked if there was any way that we could keep an eye on them.
Having just finished reading Sudo Mastery by Michael
W Lucas, I suggested that we could record their login sessions with sudoreplay
and a few setting changes. I made a couple of changes to the
sudoers file, the ~/.bashrc for the profile
that the developers would log in as, and I wrote a friendly little
script that would allow me to view their sessions based on various
different sort criteria. When they started to log in and do their work,
every keystroke and every character printed to the terminal was
faithfully recorded. Looking back at my work now, it was very janky. The
bashrc would first check if it was running inside of sudo,
and if not it would start a new shell with sudo. Sudo was configured to
allow the user to do this without asking for a password. All of the
replay files got stored on an NFS (No Fucking Security) share that was
accessible on all of the servers. However, for all its faults, my script
did one very important thing: it got the job done. This
is always the most important thing in computers. Code quality,
infrastructure reliability, and user-friendliness are all important, but
nothing trumps functionality. Functionality is the gold standard. If
some tool or setup doesn't do the job that it needs to do, then it is
useless. If it does the job while being poorly designed, it is better
than not doing the job while being well-designed. I was only pretending
to know what I was doing, and I did a job that I would now consider
sub-par. However, I got the job done, and that was the most important
part.
What I saw in the replays shocked me. The vendor employees, the
people who worked for the same company that developed the software in
the first place, the people whose primary job is to get rented out for
these exact migrations, had no idea what they were doing. I watched them
display the lowest level of general Linux knowledge and the most
copy-pasting that I had ever seen in my career. They would run the same
command multiple times, changing nothing, and see it continue to spit
out the same error messages. They would paste (non-idempotent) commands
from some document out of order and then do it again in the right order.
I watched them fail to utilize even the most basic flags for
grep and tail. I even watched them use
passwords that belonged to other companies on our systems. This last one
raised a lot of anger when I brought it up in a meeting - what if they
were putting our passwords into other people's systems as well? However,
they did one thing right: they got the job done. My
employer paid their employer nearly 6-figures.
It doesn't matter if you are only pretending to know what you are doing. If you are just pretending to know what you are doing, then you are just like everyone else. What matters is if you can get it done.
I am privileged to have a lot of non-IT friends and family, and this has given me the opportunity to see this same pattern play out in non-tech fields.
When working on cars with some friends who are professional or hobbyist mechanics, I have seen them pull out their phones and hunt down a YouTube video while underneath a car. In the thick of it, covered in grease, with the problem (literally and figuratively) over his head, I watched one of my friends pull up a video to figure out how to get to some particular part that needed to be replaced. This has happened many times. I have done it while working on my car. You have probably done it while working on your car. At the end of the day, the car runs again and drives away. The job got done.
I knew a general contractor. This guy made his living by working on house projects for others. They would pay him no small sum of money to renovate their bathrooms or put a gate in a fence. He had the expertise to do any job that came his way - he had built multiple houses from the foundation to the finish work in his life. A new client wanted some fresh flooring in a spare bedroom, and the specific type of flooring was something that the contractor had never seen before, it had been invented and made commonplace since the last time he did a floor. So what did he do? He looked up a YouTube video and he got the job done.
You cannot look into the future and see what you are going to do tomorrow. You cannot look into the future and see what you are going to do in 10 minutes. You can have an expectation, sure, but you cannot know for certain. If there is a hammer in your one hand, a nail in the other, and you are standing in front of two boards of wood that need to be nailed together, you can guess as to how exactly you are going to get it done, but you can't really know for sure until you've done it. You can know what you did, but you can never know what you are going to do. All you can have before you do something is a guess, however educated it may be.
When I need to deploy some new server software, I have no idea how complicated it will be. Probably it will just be a package installation and some changes to a configuration file. Nginx is a good example of this. However, it could require custom-compilation, various services as dependencies, extensive modification of multiple configuration files and changing settings that can only be changed graphically or through a web interface. LibreNMS is a good example of this type of setup. When I start out, I cannot know how to do something. All I can do is say how it was done after I have finished. For every new task there are unknown unknowns.
When I was in job training to become a security guard, the instructor was going over what you should do if you have to caution-tape a crime scene so that it doesn't get disturbed while you wait for police to arrive. He said that you should take "how big you think the scene should be and double it". Being a smartass 18 year old, I asked if we should double the area or double the perimeter, to which he replied "both". This is impossible. Take a square with the perimeter of 4 feet and double its perimeter to 8 feet. Now instead of having 1 foot on each side, it has 2. However, the area of the square has changed from 1sqft to 4sqft. Doubling the perimeter quadruples the area with squares. Circles work the same. Take a circle with a circumference of 4 feet and double it to 8 feet. The circle's area in this example grows from 1.27 sqft to 5.09 sqft, again a quadrupling of area for a doubling of perimeter. The instructor in this training didn't know the geometric details, he just wanted to tell people that they should go bigger than they expect because you can always make it smaller afterwrads, but a contaminated crime scene can't easily be uncontaminated. While his statement may have been wrong, it got the job done.
I was having some trouble with Ubuntu's new autobuilder, Subiquity. I actually wrote a whole blogpost about it here, but long before that blogpost I was having issues with the partitioning setup. The partitioning needs to be different for BIOS vs UEFI and it needs to have a bunch of special things set that aren't documented anywhere. I wanted to deploy different systems with different partitioning schemes and different amounts of SWAP space, but the Subiquity auto-partitioner was just too confusing. Even when I manually partitioned a system and then recovered the Subiquity-generated autobuild file, it was so opaque and specific to that server that I couldn't get my desired partitioning scheme to work on other servers via Subiquity. My solution to this was to go with the absolute default behavior of the Subiquity auto-builder, which is to create a 50GiB root partition and leave the rest unprovisioned. Since I had to do this to dozens of servers, I wrote a SaltStack state to do the rest of the provisioning. Salt doesn't include this functionality by default, so I had to make it myself. This required a lot of custom Python for acquiring information about the disk and doing math to calculate how big different partitions should be in order to meet my requirements. This was terrible, janky, bad code. I disliked this when I was doing it and I regret it now. After a few months of using this, I hunkered down and spent the 3 days that I needed to fully understand Subiquity's auto-partitioner through a mixture of trial&error and documentation. I was wrong to do it in my original manner and I wasted a bunch of time. However, I did in fact get the job done. For a few months, my custom solution was used to build out a large number of servers.
Everybody is human and makes mistakes. You need these failures in order to succeed. This is unintuitive. You don't make a good meal out of bad ingredients. You don't build a good philosophy out of bad presuppositions. You don't make an awesome castle out of broken Legos. You can't polish a turd. But failure is a prerequisite for success. Failure gives you knowledge and perspective. That knowledge and perspective gives you success. Success is when you get the job done.
Of the ~5 people who might read my blog, I don't think that any of them often experience imposter syndrome. Maybe someone will find this through a web search in 10 years. Maybe it will help them.