This might sound like a joke at first. But if you think about it, In these days, we tend to celebrate the Michael Scotts of the tech world - the enthusiastic early adopters, the ones who hype every new tool and move fast. But in the age of LLMs, I'd argue the engineer you actually need is Dwight Schrute.

Dwight Schrute would be an insufferable person to work with in the age of LLMs. He'd question every output, verify every claim, refuse to trust anything the model says without cross-referencing it himself, and probably have a 40-page threat model for your company chatbot. And honestly? That's exactly the kind of engineer I want on my team right now.

Let’s dive a bit deeper into each side of his character.

Dwight questions everything.

One of the good characteristics of Dwight is that he trusts verification over assumptions; he doesn’t take stuff at face value. If an LLM tells him information about something, he would verify it either with his own knowledge or his great research ability. You know who would take anything that an LLM gives without a second thought? Michael!

If you remember, Dwight has his famous word “False” when he fact-checks something. And that instinct is exactly what working with AI requires. LLMs are famously confident even when they’re completely wrong. The major labs are working into decrease the hallucinations in these LLMS but that problem is still not solved. For example, they’ll cite papers that don’t exist, write code with subtle bugs baked in, and summarize things from the sources you provide even in the times that they couldn’t open these sources.Every output deserves a “False” until you’ve verified it yourself.

Michael would screenshot the response and paste it straight into the proposal. Dwight would fact-check it first.

Dwight understands the fundamentals

Beside being a top employee at Dunder Mifflin, he has side hustles that he operates with deep knowledge. For example, the Schrute farm, that he operates with deep expertise in farming. He has always the urge to learn things in depth and to be competent in them. This is key now when dealing with mass production machines like LLMs. Understanding the fundamentals makes you able to better drive and use these tools. If you are good at your craft, these tools will help you magnify that; if you are bad, it will magnify that as well.

Dwight is skeptical of hype.

In an industry that is drowning in AI hype, with all of these headlines saying AI will replace everything, Dwight will not buy this without looking more into it. He would be the total opposite of Michael in this situation, who will believe any hype and act on it as soon as he hears it.

Dwight is always prepared (sometimes too seriously).

Dwight's threat assessments were absurd, but the instinct was right. In the LLM era, prompt injection, data leakage, and model misuse are real attack surfaces. Dwight would be prepared for that, just as he was prepared for all the worst-case scenarios that could happen in the office. He would understand that as his work and systems, and daily routines started to rely on a third-party AI API, he needs to be prepared. And using his knowledge and abilities, he would be able to carry on if these tools failed to operate or started to fall behind in quality.

He refuses to be replaced without a fight.

When his job was threatened, Dwight didn’t panic or give up. He made sure to double down on what made him unique and valuable. This is a great trait that we need to look deeply into these days. AI is automating the generic; if that is all you have, then you need to search for your Schrute Farm equivalent or your deep expertise and Dwight-level depth of experience.

The broader lesson Dwight teaches is that being genuinely, deeply competent in a rigorous way is what survives technological hype cycles and disruption. He’s not the coolest person in the room. But when things actually go wrong, everyone needs Dwight.

An engineering workflow mostly goes like the following

1. Concept

2. Design

3. Development & Testing

4. Commit & Push

5. Integration and regression testing

6. Deployment

The longer a problem stays undetected, the more more costly it becomes to your system, team, and the product.

This is why in serious development life cycles and teams there is focus on tools to catch various types of problems as early as possible. Something like Pre-Commit hooks, linters, type checking, tests, and CI pipelines enforces the same standards and safety every time a change being added. Of course it would be perfect if we even can catch these issues before that.

I even argue that this matters even more now. With the current advancement of the AI tools, the speed and quantity of producing code started to surpass our ability to read and review the code with keen and careful eyes.

This situation requires us to put more time and effort into creating well established, strict guidelines and quality checks more than ever. This can be started with pre-commit hooks that watch both the correctness and style and carry through to the CI/CD.

Aside that, you also need to put guidelines on your code generation machine which is one step upstream. Right now, you can setup agent rules and instructions in every major AI coding agent available. Many are converging on shared conventions like AGENTS.md. A step further would be to create a separate agent that can read and review the generated code from your code generation agent and thus help you with finding problems at the same speed of the code generation.

This is a new era of the software development tools. AI reduces the cost of producing code, but it can increase the cost of validating it unless you move validation earlier. Treat your quality checks as part of the product: explicit standards, automated enforcement, and feedback that arrives while developers still have context.

As I grow in my career, I look more and more for ways to learn from experienced engineers and from their journeys. One way is by asking people in my circle for advice and perspective. For those I do not know personally, I rely on what they share online, books, or conferences.

While I look for content focused on experience, philosophy, and the nontechnical side of engineering, I have found that writing about engineering maturity is harder to find than technical tutorials.

What is maturity, and how do we define it in engineering? Is it simply being a senior engineer with a lot of system knowledge, or does it require something else?

To be honest with you, I don’t know the full set of requirements for being a “mature engineer”. But I’d recognize certain traits when I see them. This set of traits I look for in my teammates, and in candidates I interview.

Admit what you don’t know

One of the clearest signs of maturity is the ability to admit when you do not know something, or when you need to research and learn more. In my experience, this quality is a strong signal of maturity. Being a senior engineer does not mean you know everything in your field. Admitting what you do not know does not diminish you, it often earns trust. For me, this is one of the most important traits I look for in someone I want to work with.

Seek Constructive critique early

One trait I try to develop in myself, and that I see in many great software engineers I respect, is actively seeking constructive criticism. When I design a system, I share it with peers, provide context and background, and encourage them to challenge it and poke holes in it.

This is not because I do not trust my abilities. It is because good systems are hard to build from a single perspective. Even with experience, I am still vulnerable to bias, and that bias can show up in my designs. Constructive peer reviews often improve the design and lead to better decisions.

Also, if you are proposing something, make sure to study it well and have an argument you can passionately argue for it. Being passionate, prepared, and genuinely interested is an admirable quality!

Be a Multiplier for others

Multipliers elevate their colleagues. Do you help less experienced engineers learn and grow? Do you make the people around you better? This goes beyond being good at your craft. It elevates the team and increases its impact across the organization.

Respond well to mistakes

I believe that making mistakes is part of being alive. Learning how to deal with mistakes is part of growing up and maturing. In software engineering, mistakes happen. We try to catch them before they reach production, but some will still slip through.

I have made my fair share of mistakes throughout my career. When I reflect on how I handled them early on versus how I handle them now, the contrast is stark.

Back then, I tended to panic, and sometimes I would simply stop thinking straight. I focused on managing the blame (which I often just imagined) rather than the issue at hand. As I have grown, I’ve learned to replace that fear with structure. Now, I prioritize mitigating the damage and fixing the problem first, then calmly investigating the cause later. This shift from ‘fear of blame’ to ‘focus on solution’ has been a major part of my growth.

Mature engineers handle these moments effectively. They stay calm, focus on fixing the issue, learn from it, and help the team move forward. Later, when it is safe, they can even laugh about it and move on without blame.

Understand the importance of context

This is something I touched a bit in my previous blog. Context is the key component when proposing a solution, rather than blindly throwing out one-liner rules or principles. Knowing lots of terminology, principles, design patterns is great, but learning that “best practices” are not always best. You need to learn about the context of the situation and take it into consideration when you decide or propose something is such an essential part.

Some of the hardest decisions are the compromises based on the current reality of your team, organization, project, and timeline.

Have a business awareness

Understanding the code and the system architecture is very important and crucial for the engineering team. However, adding business awareness takes you to another level. Asking “why are we building this product?”, “who are our customers?” and “what does the customer need most?” brings more understanding from the product side which increases your effectiveness and helps bridge between the deep technical understanding and the product understanding.

Read the room and communicate for your audience

Earlier in my career, when I was a junior engineer, I used to go deep into technical details with people who were not interested in them. Then I would complain that they did not understand the background, and I would naively wonder why they were not interested in those details. That version of me was missing the point.

Another crucial part is understanding that higher-level managers are rarely interested in specific technical details. They are interested in the product’s development state, how it affects the timeline, and how we are mitigating risks.

Having the self awareness when it comes to communication takes your presence and room reading into another level. Being intentional about how you deliver opinions and criticism, learning how to present to each group, what they are interested in and what parts that they aren’t, makes you able to reach to multiple audience.

Adapt calmly to change

Personally, I’m not a big fan of lots of changes happening, I love to have systems around me. I usually sleep at the same time no matter it’s a work day coming or a holiday. I wake up at the same time. have a similar routine in the morning, buy the same shoes or similar if I couldn’t find them. You get the idea.

Ironically, I spent most of my career working at startups and mid-sized companies which have a higher frequency of change happening than the big tech companies (at least in general). As a result, I started to learn how to adapt to that and learn how to build my systems around it. The changing nature of requirements and focus in startups, and the need to deliver fast and iterate, makes learning how to adapt with that is absolutely necessary!

I still have some stuff to adjust and figure out in that regard. The uncertainty of change is not comfortable by nature. The main goal is not to make it look and feel better, the main goal is to learn how to react calmly and correctly when it happens. Life is built on change, it’s unavoidable.

Conclusion

These traits are not developed in a day or two; they are the result of scars, mistakes, and lessons learned over years.

True maturity isn’t about having all the answers or building the perfect system on the first try. It is about having the resilience to handle the moments when you don’t know the answer, the humility to ask for help, and the wisdom to prioritize the team’s success over your own ego. It’s not a destination you reach, but a mindset you choose every day.

I don’t know the full set of requirements for being a ‘mature engineer,’ but these are the traits that have signaled growth in my own journey. It is a mix of technical context, business empathy, and emotional intelligence.

As I continue to learn, I am sure this list will grow. I’d love to hear from you. What is the one non-technical trait you value most in a senior engineer? How do you define maturity in our field?

Further readings

• https://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/

• https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/

Best practices give you valuable insight into how the creators of a tool or framework envisioned its use. They offer battle-tested approaches that have worked well in many situations. Think of them as the collective wisdom of the community, a starting point for understanding what “good” looks like.The wrong way to read the Best Practices, is to instantly consider them the best for your context or use case. I have seen many people fall in this trap, once they read that something is best practice, they stop thinking anymore and take it as a face value without even taking sometime and examining if their usecase fit this best practice or not.

For example, they read that microservices are a best practice, so they break apart a perfectly functioning monolith. They see that event-driven architecture is recommended, so they introduce unnecessary complexity to a simple CRUD application.

The problem isn’t the practices themselves, it’s the lack of judgment, context-free adoption of them.

Context Is Key

Engineering problems, especially those involving system architecture for business needs, don’t exist in a vacuum. They live within a web of constraints and requirements:

• What are your actual performance requirements?

• Who are your customers and how do they use your product?

• What’s your team’s expertise and bandwidth?

• What’s your budget and timeline?

• How critical is this system to your business?

When you ignore these factors and architect in isolation, you risk solving someone else’s problem while completely missing your own. That scalable, fault-tolerant, highly-available system might be impressive, but was it what you needed? If all you needed was something simple and maintainable delivered quickly, you’ve failed miserably.

Best practices can be the best for some situations, maybe good for others, and doesn’t work for many. Use your brain, evaluate and choose the parts that works for your situation.

Before you decide to follow one, ask yourself:
Does this practice address a problem I actually have? Does it align with my constraints? Will the benefits outweigh the added complexity in my situation?

The process of design is similar to me to the process of writing. I can’t confirm or deny that I’m good at it. To be honest, I’m not sure if there is a definitive way to align on a certain definition of “good” in that context.

But I know the process that makes me able to draft a better version than the previous one. and that’s what I’m here to discuss.

The Process: Everything starts from a problem

1. Study the Problem

The beginning is usually simple: a big problem everyone is staring at. From the outside, it looks clear from the outside. It can range from fixing a brownfield project to creating a new greenfield one. What’s missing for most people is the detail. What are we actually facing?

Your first step: do your homework and study the problem.

Studying the problem goes beyond reading whatever document exists. You need to draft the details yourself and create a starting point—a shared definition of what the problem is, how it started, and what caused it. Understand the past and present to set the stage for the future.

Sometimes the details feel shallow and scattered. Don’t worry; that’s common. Your best tool here is documentation. The one you write after gathering every relevant piece of information.

2. Think, Digest, and Decide

Once you’ve collected enough information to organize your thoughts, start the thinking process. Put everything in front of you and start connecting the dots.

What helps me is printing the key documents and studying, marking, and taking notes with pen and paper. This might differ in your case. The important step is to connect the dots.

Imagine Dr. House sketching on the whiteboard, brainstorming with his team about possible diseases based on the symptoms.

3. Start drafting an initial design

Draft an initial design that captures your initial thoughts. It probably won’t be the final one, but it’s crucial to move forward.

This initial draft becomes the basis for all next steps; it’s the one that you will iterate on. It may be abstract and light on details. Don’t worry about that. The most important thing is to draft this design in a format that you can edit and adjust. There are plenty of tools out there that you can use, for example draw.io . Pick what suits you and your company’s style. The tool matters less than its ease of iteration.

4. Challenge the design

Before moving on, challenge the design using what you know about the problem. Does it work for basic cases? What about edge cases? When does it fail? Does it address everything you currently know, or did something drift?

If you find something that you missed, try adding it to the draft. If you find yourself stuck or can’t see what’s missing, it’s time for the next step.

5. Talk to people

You have challenged your own design with the information you had; now it’s time to talk to people and make them challenge it and give you different views. and maybe additional context that can help you evolve this design.

This step is tricky. Who do you talk to? How do you give them context?

What works for me is 1:1 conversations with people who have more experience or a different perspective—often stakeholders or adjacent teams. People from outside usually provide great insights and ask sharp questions that can help reveal blind spots.

Why 1:1s? They create focus and usually end with clear action items. They’re more controllable and productive than larger meetings. Don’t get me wrong, the wide audience meetings are useful but mostly for later stages. From my observations, once you exceed 3–4 people in a meeting, it gets harder to steer and often end without clear outcomes.

I prefer the wide range one in the latter stages of the design, when things become more solid, and I need alignment with the wider team on the final touches.

6. Take the feedback and repeat

After challenging your draft and talking to people, collect the feedback and update the design to the next version.

As you repeat this loop, watch and observe how your understanding of the problem and your solution evolve. That evolution is one of my favorite parts: witnessing your steps toward the solution.

This process may seem simple, but it requires effort and the right mindset. The key here is Don’t get attached to your design, and don’t take anything personally.

The goal is to build the right feedback loop and iterate until you reach alignment with the objective, the problem, the primary and edge use cases, and the stakeholders.

7. It’s time

Here you start the journey of implementation. At this stage, If you iterated enough, changes should be minimal. There are cases when you will find changes needed during implementation. Document them and ask yourself, what did you miss? Why did this appear now and not earlier?

This will make you evolve as well. It’s your feedback loop driven by actual experiences in the battlefield.

Good design isn’t a one-shot thing; it’s a loop. Study, draft, challenge, talk, iterate, ship—then learn and repeat.

Test-driven development is a way of writing code that involves writing an automated unit-level test case that fails, then writing just enough code to make the test pass, then refactoring both the test code and the production code, and repeat.

> https://en.wikipedia.org/wiki/Test-driven_development

One of Test-Driven Development (TDD) main goals is to make sure you write tests and don't skip them. While this goal is good, I don't think TDD is the best way to achieve it. Why?

Strategic Design vs. Test-Driven Tactics

Software projects, especially medium to large ones, need strong designs to handle inevitable changes. To do this, we begin by strategically designing the project.

Software design looks at the whole system first, then digs into how each part will work together. This requires thinking about the structure (big picture- Stratigic) before digging into the smallest parts (which the unit tests are concerned about). As design ideas mature, the structure becomes clearer and more detailed.

Software Design is a continous process, it usually starts with diagrams, RFCs, and (God forbid) UML. As you explore the problem deeper, more details emerge, you find another aspect that you need to take into consideration. This growth isn't always smooth, but it generally expands over time. During this evolving process, code is always being written, changed, and reused. The TDD approach would make us less adaptable, slowing us down and hurting productivity.

TDD can be showcased as a guide for small-scale design decisions, and in some reallife situations requires you've already or mostly solved the problem in your head (even if it’s in theory says it’s not). It’s “test-first” requirement is not suitable when the design is bigger and changing in scale.

With over seven years in software (which is not much, I know, but also not little), working on various projects for many companies, I can barely recall a single project that was fully figured out or even most of it from the start. Things constantly changed. Starting with tests first makes the primary goal of development simply passing tests. The focus shifts to getting "green checks" on the screen, losing sight of solving the overall product design.

I Still Like Unit Tests

That said, I love unit tests! I use them to make sure to protect my code ( or at least a considerable part of it) from the mysterious future, and immature changes.

And yes, they aren't much fun to write (at least for me). But I don't need to enjoy them to see their value. My favorite part is when a test I wrote fails after I mess something up during coding. It's a great safeguard, preventing minor mistakes from becoming major production outages. I'd much rather see red text on my screen than a massive problem for my silly bugs.

For me, unit tests typically emerge after the initial code for a module or feature has been drafted, or sometimes in very tight cycles alongside the code as its design solidifies, but never as the initial driving force.

I like my development to be drivin by strategy not green text.

No, this isn't another article about the advances in LLMs. As much as I'm interested in the AI space, I'm far more concerned with human intelligence and keeping my own mind as sharp as possible.

Throughout history, humans have used tools: knives to cut, cars to travel, calculators to compute, and pens to write. The greatest invention of the last century, however, was the computer. It's a phenomenal machine that gives us the ability to innovate, transforming our thoughts into executable creations. This magic was amplified when we connected computers, creating the internet and unlocking their seemingly infinite potential.

Every invention requires new skills to operate it in return for its power. You learn to drive to travel far and take others with you. You learn to type to harness a computer's computational power and access a universe of applications through its vast network.

Until now, every app and invention has focused on providing an advantage in a limited domain. A car promises to take you from one place to another. Computers are more versatile, but they are still limited to what you explicitly tell them to do; they don't hold knowledge on their own.

LLMs are different because they use language as their primary interface—the very tool we invented to communicate and formulate thoughts. Languages are designed to be only limited by their user. The more words you know, the more ideas you can communicate.

An LLM is designed to be powered by text, a huge amount of it, with a huge number of examples in different shapes and forms. Unlike other tools, It doesn't confine itself to a specific domain. It's designed to answer any question on any topic, as long as it's phrased in a language it has seen before.

To do it justice, it's one of the best search tools available, in my opinion, not only fetching what you asked for but also reformulating it in styles and shapes that didn't exist in the original sources.

When you look at it, It behavior feels like that person we all know who answers every question with supreme confidence, but without any process for verifying the accuracy or factuality of their statements.

Submitting to it completely will mean erasing your ideas and replacing them with its output. This will not only change what’s is being presented at the moment, but it also stops any chance of you evolving your style, ideas, and mental muscles. You will be binding your thought process to a cloud-based service that changes over time and is controlled by one of the big companies that has zero connection to who you are and what makes you, you!. Is that something that seems pleasant to you?

This is where you must stand guard. Your ability to distinguish fact from fiction depends entirely on your own critical thinking. If you blindly accept what an LLM provides without being aware of its limitations, you risk degrading your own intellect and losing your edge. Keep your eyes open and your mental muscles trained. Use these tools to accelerate your work, but always be the one at the gate, checking what comes through and securing your mental castle.

1. Your Core Tech skills

Every role demands a specific foundational skill set. For a Software Engineer, this includes coding and system design. For a Chef, it's cooking; for a Driver, it's driving. These core tech skills shine as the main ingredient, especially at the beginning of your career. As they hold the keys to entering the domain

2. Product management

Nope, this isn’t about the meaningless, endless meetings discussing hypotheticals and wasting time. I mean here the true product management, the one that involves consistently maintaining a clear vision for the project: understanding precisely what needs to be done, why it's important, where the next steps should lead, and how to get there effectively.

3. Project management

Sometimes, it gets tricky to differentiate it from the previous point. I don't blame you.

The main objective of this skill is one word, "Delivery". removing obstacles from the way to ensure that work is completed and key milestones are reached.

4. People management and skills

Building a successful team is not as simple as gathering employees under one name. The more senior you get in your career, the more you need to think about how to make the team work efficiently and how to help them grow. This involves understanding their needs, communicating effectively, and helping to resolve challenges.

As you advance, the importance of these four ingredients becomes increasingly clear. The expectations from you as a senior get broader, requiring you to fluidly apply these skills in varying proportions depending on the specific situation. Fluidity is key.

DoS (Denial of Service) is an attack that overwhelms a resource or service with a ton of fake or meaningless requests. Imagine a website suddenly getting flooded with so many empty requests that it can't handle the real ones from actual users. It either slows down to a crawl, denies legitimate access, or completely crashes, making it unusable.

You can achieve this in the tech world by setting up a job that repeatedly calls a resource a massive number of times all at once. This pushes the resource past its limits (whether by scaling up or out) and makes it slow down, increasing the time it takes to respond. Eventually, it becomes unusable for real users or even stops responding entirely.

How to DoS Humans:

This attack isn't just for software and servers; it applies to people, too. We all have limited energy and capacity. If you want to "DoS" someone, you essentially flood them with so many requests and tasks that their time and mental capacity are fully occupied. When this happens, they start falling behind on important tasks, can't respond to new requests, and quickly burn out.

How to unintentionally DoS your team:

I bet you're seeing where I'm going with this. DoSing your own team can happen quite easily, even without you realizing it. In the past, common ways to unintentionally overwhelm your teammates included:

• Asking your teammates about every little thing before even trying to find the answer yourself.

• Scheduling a ridiculous number of unnecessary meetings in the name of "super duper scrum."

• Sending really long messages about something that could have been said in just a line or two.

But now, there's a new set of "DoS" attacks emerging, especially with the rise of AI tools:

• Submitting Pull Requests that are entirely AI-generated without you even reviewing them properly.

• Generating huge, complicated chunks of code that are a nightmare to read, maintain, and review.

• Writing long, AI-generated messages that sound professional but totally lack your personal touch and clarity.

These actions can seriously exhaust your team's capacity. They'll perform less, become less tolerant of your future requests, and this negative impact will eventually loop back and affect you too.

Be responsible:

I'm not trying to be a hypocrite here, telling you to stop using AI in your work!. I use AI in my work. My point is to use it responsibly and always keep your hands on the steering wheel. Don't just hand over your brain and judgment to the AI. And most importantly, don't confuse generating more code faster with being truly productive. Productivity is a mix of both quality and quantity. If you lose quality, you are just producing lots of bad code very fast.

Stop DoSing your team and try to use these new powerful tools in the best responsible way you can.

Technical debt can be tough to wrap your head around. In its core, it’s like any kind of debt. It has interest that represent the cost of borrowing!

In conventional debt schemes, you borrow money that you don’t have to afford something that you normally can’t afford on your own at that moment of time.

Imagine using a credit card to buy a new laptop. You get the laptop now, but you'll pay interest later. Technical debt is similar: you get the feature out quickly, but you'll pay for it in increased maintenance and bug fixes later.

The Cost of Rapid Growth

Technical debt manifests differently depending on the company's context and stage. In fast-paced startups, it often arises from the need to quickly adapt to changing requirements. There is always the push for shipping new things and changing requirements and directions.

Usually a small to medium team of talented engineers is hired to convert the vision of the company from words and logos into products, user-base and revenue.

When you work in a rapidly changing and growing field, change is a main actor. When priorities shift, engineers have to scramble and readjust their plans, sometimes cutting corners in design and implementation that introduce technical debt.

This approach can work in the short term to meet current timelines, but it will bite the business later, either by breaking or by slowing down any further development, which can affect the business's fate and finances badly if not properly addressed

How Ignoring Tech-Debt Can Affect Non-Tech Manager

Let's explore how technical debt, often seen as a technical issue, can impact the real-world problems that non-technical managers care about

A typical timeline would be like this:

Initial Change:

The company changes timelines and wants to release something earlier than intended. Executives inform managers about the updated plan and ask them to make it happen. Managers reach out to tech leads and ask them to expedite the process. Tech leads try to explain the cost and drawbacks of releasing things earlier than planned, mentioning technical debt briefly. Managers, focused on immediate goals, often overlook these warnings.

Short-Term Success:

After a few months, the product is successful and needs to grow more, adding new features and building new products on top of it. Tech leads explain that they need to address the technical debt generated by the first iteration to confidently build on top of it. Ignoring this might risk the whole project becoming unusable or requiring a huge re-write.

Long-Term Consequences:

Managers, recalling that ignoring technical debt warnings worked out fine the first time, decide to ignore them again. This time, the product updates are shipped with more problems than new functional features, and fixing them takes more time than before. As a manager, you face problems with the engineering teams who tried to warn you and with the upper management and executive team because you are unable to deliver on promises and manage things properly.

All of this add up and compiles into

• Decreased development velocity

• Reduced product quality

• Engineering team frustration

• Missed deadlines and commitments

• Loss of trust from upper management

• Increased maintenance costs

Dealing with technical debt as a non-technical manager requires understanding how it can affect the entire chain. Ignoring it is a recipe for failure.

A classic example of the impact of technical debt is the Southwest Airlines shutdown. The airline's failure to address technical debt in their crew scheduling software led to losses exceeding $1 billion and severe reputation damage [source]

But how should we deal with it? No no, how we can prevent tech debt from the start?

You can’t prevent technical debt entirely, and you shouldn’t think this way.

The Good Side of Technical Debt

Technical debt is often seen as a villain that eats time and effort that could be used to build useful things instead. But this is not the full story.

Technical debt is a natural result when you need to move faster and deal with real-life conditions of moving and changing environments. It allows you to borrow from the future to achieve things in the present like

• Faster initial delivery

• Market opportunity capture

• Flexibility in development (in the short term)

• Creates space for real-world adaptation

• Can be a strategic tool when managed properly

It becomes problematic when you ignore the increasing interest and don’t pay it off as soon as possible. Without this ability, your business can miss opportunities that might cost a lot.

A good management of tech debt requires good engineering and leadership. Learning when to go for it and when to stop a bit to take a breath and fix things.

Dealing with Tech debt is not a luxury; it’s a necessity!

How to deal with tech debt in a healthy way?

Dealing with technical debt requires a collaborative effort between management and engineering.

Engineering

Engineers need to keep things simple and maintainable. They should track technical debt and allocate part of their time to keep it minimal. Tech leads need to keep things in check and communicate the full status to managers, informing them about the team’s needs and challenges.

To effectively measure and manage technical debt, consider using metrics like code complexity, code churn, and defect density. Tools like SonarQube and CAST Highlight can help you track and analyze these metrics.

Also, good coding practices, regular code reviews, and automated testing can significantly minimize its accumulation

Management

Managers need to listen to technical leads and not ignore their updates and advice. They should keep the team updated on potential changes from the executive team and work on setting reasonable expectations that the team can meet, along with a plan to minimize future issues.

• Prioritize technical debt repayment during sprint planning.

• Allocate specific time for addressing technical debt.

• Foster open communication between engineering and management.

By understanding the strategic value and potential risks of technical debt, non-technical managers can work collaboratively with their technical teams to ensure project success and long-term business health.

Humans are curious by nature—it’s part of what drives us to learn and explore. When we look up, we see a mysterious sky filled with the sun, moon, stars, and endless darkness. To satisfy our curiosity about the mysteries above, we had two options:

1. wait patiently until we evolved wings

2. invent ways to reach space ourselves.

While I'd love to have a pair of wings and strong lungs, I certainly wouldn't bet my life waiting for such evolution!

In the last century, human civilizations raced each other toward the dark skies above. We built rockets, satellites, and spaceships. Some were more successful than others, but nearly all played essential roles in writing humanity’s history as we broke the barrier between our planet’s surface and the stars beyond.

I’m here to explore with you our journey (so far) to our next step. We've already built swarms of eyes around our planet to gather data—it's now time to add intelligence to them.

The History of AI in space is not long, but not shallow as well, we are still in the beggining of it. let’s take this as a chance to reflect on

• The AI Applications in the space industry

  • On board

  • On ground

• Challenges and future

Let’s dive in

AI applications in the space industry

Think of satellites as flying computers equipped with sensors, floating wirelessly in space. Since we can't physically sit beside them and watch their screens, we need a mechanism to interact with the data they collect. This interaction happens through "downlinking" and "uplinking." Downlinking occurs when satellites send data down to our ground stations, while uplinking is when we send commands or data back up.

Unfortunately, this process isn't as fast as your home WiFi—it requires complicated equipment and takes considerably longer.

Currently, satellites collect data much faster than they can transmit it back to Earth. We face a gap between onboard computational power and communication speeds. So what can we do about it?.

Here comes the first application:

Onboard AI Applications

Smart Filtering and prioritization for data collection

In November 2000, NASA launched the Earth Observing-1 (EO-1) satellite. EO-1 showcased an onboard AI software called the Autonomous Sciencecraft Experiment (ASE). ASE analyzed images onboard and filtered scientifically interesting events—like volcanic eruptions, floods, and wildfires. This dramatically improved data collection efficiency.

More recently, ESA's Φ-sat-1 (launched in 2020 as part of the FSSCat mission) used onboard deep learning models to detect clouds in Earth images. By doing this, the satellite only transmitted clear and valuable images, significantly saving communication resources.

Smart Navigation and Autonomous decision making

Another great application of the on-board AI applications on satellites is the navigation. Satellites are like semi-self-driving cars that drive itself (most of the times) on the roads (we call it orbits). As the space around us has many other objects coming and going, some of them we don’t control, like astroids, others we do, like other satellites. It’s not totally safe out there and we need a smart way to handle that.

NASA's OSIRIS-REx mission employed a system called Natural Feature Tracking (NFT) to autonomously guide the spacecraft to asteroid Bennu's surface for sample collection. NFT allowed OSIRIS-REx to achieve a precision landing within one meter of the target area—a remarkable feat of autonomous navigation.

More recently, ESA's Hera mission (launched October 2024) is another example. Hera autonomously navigates around the Didymos asteroid system, using onboard AI to map surfaces, adjust orbits, and perform close maneuvers—all without continuous ground control intervention.

Satellite Autonomous Operations

Usually, satellites get to know what it should do at a certain point of time by tasking. The ground stations send information about what tasks to do and at which order.

NASA tested a new approach with its ASTERIA CubeSat, using multi-mission, goal-based, onboard, integrated planning and execution software called MEXEC (Multi-mission EXECutive). MEXEC leverages Hierarchical task network (HTN) planning is an approach to automated planning in which the dependency among actions can be given in the form of hierarchically structured networks.

AI Specialized hardware on spacecrafts

Companies have recently begun developing low-power, radiation-hardened AI processors designed specifically for satellites. For instance, KP Labs' Leopard System, A CubeSat-compliant Data Processing Unit (DPU) utilizes Deep Neural Networks for onboard data processing. With a throughput of up to 3 Tera Operations per second, it enhances the satellite's ability to process data autonomously, improving efficiency and reducing the need for extensive ground-based data handling.

AI on the Ground

Moving back to the ground side, we will find a variety of applications waiting for us, starting from the ground station and mission control and moving towards the satellite data processing and modeling

Mission Control and Ground Station Operations

Mission Control Centers (MCCs) are facilities that manage space missions from launch to landing or mission completion. They serve as the central hub for monitoring and controlling spacecraft, ensuring mission success through real-time data analysis and decision-making. If you've watched any rocket launch online, you've seen them—they're the rooms filled with screens and engineers monitoring live data.

One example of the AI applications in this area is NASA's ASPEN Mission Planner. An AI-driven system that optimizes mission efficiency by automating complex scheduling tasks and resource allocation for various space missions

Earth Digital Twin

Imagine gathering all of the earth observations that we are collecting and creating a model that understands the past of the planet, watches the present and predicting the future?

This is one of the projects that ESA is working on. An AI-driven Digital Twin of Earth, Continuously fed with Earth observation data, this digital twin could help us visualize and forecast natural and human activities on our planet.

Weather Monitoring and Forecasting

When you open a weather app on your phone, you see current weather conditions and Forecasts across the day and the week.

Traditionally, these forecasts rely on physics-based models built upon fundamental equations of fluid mechanics and thermodynamics.Utilizing numerical techniques to solve complex partial differential equations (PDEs). However, these models are resource-intensive and require supercomputers or High-Performance Computing (HPC)

Today, Companies and institutions are exploring deep learning-based weather models. These AI Models capture complex, nonlinear relationships without explicit physical equations and learn patterns and relationships from historical weather data. potentially matching or surpassing traditional methods at a fraction of the operational cost. Although, the machine learning models requires a big amount of computation powers in the training phase, It require much less while running it, which eventually should be much cheaper to run and maintain than the Physics model.

Companies like Spire Global are already creating their own AI-based Weather models utilizing their Earth Observation data like GNSS Radio Occultation (RO) and other data to make it as good as or even better than the traditional models

Institutions like NASA, and ECMWF are investing as well in that area. This kind of investment will help pushing the boundaries for a better weather forecasting which is a crucial element in all of our lives

Challenges and the Future

While AI offers great promise, integrating it onboard satellites isn't without difficulties. Missions like ESA’s Φ-sat-1 face challenges such as limited computational power and tight energy budgets, highlighting current technological constraints.

But challenges indicate room for growth and innovation. we can see institutions like ESA putting roadmaps for AI utilization in their projects and missions. Meanwhile, private companies increasingly recognize AI's potential to revolutionize their business models, driving further innovation.

We stand at an exciting crossroads. As AI technology rapidly evolves, the space industry will surely follow—propelling humanity toward ever-greater exploration and discovery

"What a day!" your friend sighs, sinking into the sofa. "It began with me nearly missing my train—I had to sprint like a maniac, and everyone stared at me like I'd lost my mind. Then, at work, we had an essential meeting with a crucial client we've been trying to land for months. Usually, he's enthusiastic and engaged, but today, he seemed distracted and uninterested. Definitely not my best day."

You're listening carefully, genuinely trying to give your friend your undivided attention. Yet, as their words flow, you notice something strange happening: every detail your friend describes is replaying in your mind, but with a different voice—yours. Without even noticing, you're creating your own version of the story. And here's the irony: neither your friend's narrative nor your interpretation fully matches reality.

Stories have countless shapes

Stories have countless shapes, depending entirely on our unique perspectives. It's similar to that classic optical illusion you might have seen—the image that can appear as either a duck or a rabbit, depending on who's viewing it. But life is far more nuanced and complex than a simple drawing.

Think about how differently two coworkers might describe the same meeting: one might say, "That went horribly, the boss clearly didn't like my idea," while another might recall, "What a productive meeting! Everyone was really engaged." The actual meeting didn't change; only their perceptions differed, shaped by their past experiences, feelings, and viewpoints. Or consider how siblings can recall completely different childhood memories of the same family vacation—one remembers laughter and adventure, and another recalls boredom and discomfort. Both stories are true, yet neither fully captures reality.

We are not our Stories

We are not our stories. Our perception doesn't just shape how we see external events—it profoundly influences how we see ourselves. Imagine someone complimenting your work: one day, you might take pride and confidence from the praise; another day, feeling insecure or doubtful, you might dismiss it as mere politeness. The compliment itself didn't change, but your perception—the story you told yourself—did.

Learning to recognize that each of us carries these unique lenses, shaped by our emotions, beliefs, and past experiences, helps us gain compassion and understanding. Acknowledging different perceptions and trying to look at the world from their view expands our world, making it richer, deeper, and more connected.

Looking through different lenses

Pause for a moment and imagine all the different perspectives around you: your own, your parents', your coworkers', your friends', the barista who served your coffee this morning, or the stranger with whom you briefly made eye contact on the train. Every single person you meet interprets the world uniquely, creating their own narratives—some similar, others vastly different.

The next time you're having a conversation, especially one that sparks strong feelings or reactions, ask yourself: "How else could this story be told? What might I see if I looked from another angle?" Remember, you are not your story—your stories are merely glimpses through your lens. Reality is larger and far more complex than any single perception can capture. Learning to leave our perception for a second and examine our stories using different ones is a great skill.

Don't limit yourself to a single narrative. Step outside your story and look again. You might be surprised by what you discover.

Imagine you're working on a project as part of a 5-person team. You’ve broken the project into milestones, features, components, epics, and stories and set an estimated timeline. Everything seems planned out—until one of these scenarios happens:

1. A team member leaves the project.

2. The company faces challenges and needs the project completed earlier than planned.

3. The project scope expands, with more features being requested.
   

If you analyze each scenario, you’ll notice they each impact one of three critical factors:

1. People

2. Timeline

3. Scope

When one of these factors changes, it inevitably disrupts the balance. To address the issue, you have to adjust at least one of the other factors. Let’s break it down:

- Fewer people mean the work will take longer to complete.
- Less time requires reducing the scope or adding more people to meet the deadline (though this isn’t always effective).
- An expanded scope may require either more time or additional team members.

Why Adding More People Isn’t Always the Solution

You might wonder: Why not just add more people to solve the problem? The answer lies in Brook’s Law, which states:
“Adding manpower to a late software project makes it later.”

Adding new team members, especially late in a project, incurs hidden costs. Time is required for onboarding, aligning with the team, and catching up on the project context. By the time these new team members are fully productive, the project might already be near completion, making the payoff minimal.

Mitigating These Challenges

One way to reduce the impact of these scenarios is to split the project into smaller, independent components that don’t rely heavily on each other. This way, team members can work in parallel with less dependency. However, achieving this level of separation is often easier said than done, especially early in the planning phase.

If these challenges resonate with you, I highly recommend checking out The Software Engineer’s Guidebook. The image I’ve attached comes from this great read

Let me know your thoughts—have you experienced similar scenarios in your projects?

A usual scenario in your day to day is to get asked this question

How long will it take to finish this?

For some people, the question seem straight forward, especially the people that are not building this product. But for you as a Software Engineer (The Builder), It’s one of the most hard question to answer.

In this article, I’m exploring more on that front, exploring a methodology that was presented in The Pragmatic Programmer book.

Where Do Estimates Come From?

The process of estimating the time required for a specific task is data-driven. The accuracy of these estimates depends mainly on the data's quality and the task's understanding.

• The nature of the problem

• The nature of the available resources

  • Experiences and Team Capacity

  • infrastructure

  • Time

  • Money/Budget

• The outside factors (unplanned, uncontrolled surprises)

To create an accurate enough model, you need to consider these factors. Let's break down each one of them

First, understand the problem better than your customer

The customer here is not just someone from outside. The customer can be a CEO, a manager, or anyone requesting the task. Often, people don't fully understand what they are asking for. But you need to know and understand it well!

Start by asking questions about the requirements and scope of this task. What are the expected outcomes, and what use cases need to be covered?

Asking these questions and gathering information about the problem should help you start working on a mental model of it.

Sometimes, people ask for X, but after they explain what they want, you find out they only need Y. Figuring this out early is crucial for making accurate estimates and avoiding wasting time and resources on the wrong thing!

Understanding the nature of the available resources

Any task requires some sort of resources, for example

• Human resources: people working on it

• Architectural resources: Servers, Databases, ..etc

• Time: The time that will be dedicated to researching, developing, and maintaining this problem instead of another one

• Money: the cost of all of that to get this done and how that align with the budget

In a perfect world, we would have an infinite supply of these resources. But as you already know, our world is far from perfect! After understanding the problem, you can start assessing how many of these resources you need and how many you have. It's also important to consider the big picture. How will this affect your other projects and commitments? Make sure to communicate this to the project manager and the team or department manager.

Taking the bad day into consideration!

After you assign a reasonable value for each parameter and feel confident in your initial estimate, you need to consider a buffer for unexpected events. I know it's hard to plan for these, but they happen often.

Always add an extra buffer; you will need it. The only way to get better at estimating this buffer is through experience. At least, this is the only way I know! If you know a better way, I would love to hear about it! Please!

Adding all of this together

After you go through these three steps, gather the information from each and use it to make an estimate.

One method we can use is called the "Program Evaluation Review Technique" or "PERT." It works like this:

"So now, How long will it take to finish this?"

"Well. If everything goes right, and we are sure that we covered all the requirements and usecases, this should take from the team having Member1 and Member2 focusing on it about 1 sprint (2 weeks). But that's unlikely the case, So I would guess a more realistic figure is close to 2 sprints (4 weeks). And of course, if we happen to have a huge blocker or urgent critical issues regarding the other operational systems that we currently have and needs the efforts of these members, we would need an extra sprint or a bit more"

As you can see, this method expresses 3 main scenarios

• Optimistic

• Most likely

• Pessimistic

With this approach, you spread the range of the uncertainty to levels, which helps to get a better estimation of the project as a whole

I know, people usually don't like this answer because it doesn't match the wall-sized charts they expect. Some think they have a formula for an accurate estimate. In my experience, the likelihood that they don't is high, and the chances that they've never done this before are even higher!

Track, Improve, Repeat

You probably went through this process, got an estimate, and found it inaccurate. Now what?

This process is ineffective if you don't track your estimates, learn from what went wrong, and apply those lessons next time. It's a feedback loop, an iterative cycle of improvement. Start somewhere and aim to get better each time!

Remember, it all starts with not rushing to give estimates the first time you're asked. Take the time to think it through. Don't forget the magic words.

I will get back to you.

Hello there! As I look back on my 20s, I realize I've gone through a lot, and it's a great opportunity to reflect and gather the lessons that have shaped me. Let's walk through them together, and remember, these are my lessons. Your experiences, goals, and values might lead you to different conclusions.

Don't blindly follow others' life lessons.

It's tempting to take other people's life lessons as universal truths, but remember, we all have unique life experiences. We all lead different lives, influenced by unique events and times. Our goals and values also differ significantly. Hence, what worked for someone else might not necessarily work for you. It's always important to reflect on advice and adapt it to your personal circumstances.

One Size Doesn't Fit All

Let's talk about learning methods. I've noticed that success isn't always about following a prescribed path. Two people can employ radically different learning strategies yet achieve the same score. If they were to swap methods, their success might plummet. This reinforces the idea that we all have distinct ways of absorbing and processing information. Embrace your unique learning style and tweak it as you evolve.

The Power of 'Why?'

Never underestimate the power of asking "Why?" This simple question can unlock deeper understanding and insight. The backstory behind a concept or decision often reveals the context and factors that influenced it. However, it's important to strike a balance - don't barrage people with "whys" without doing your own research first.

A healthy Mind is in a healthy body

Prioritize your physical health. It's the foundation upon which all other aspects of your life are built. Even something as simple as a regular walk can significantly improve your well-being.

Approach New Topics with an Open Mind

Approach new topics or concepts with a fresh, unbiased mind. It's easy to carry preconceived notions and second-hand ideas into our learning process. However, these can sometimes obscure the truth. Instead, strive to build your own perspective based on your experiences. This doesn't mean dismissing experts' insights, but rather understanding their perspectives fully - the what, how, and why - before incorporating them into your own worldview.

Don't compare yourself with others, but yourself

Stop comparing yourself to others. It's crucial to understand that everyone's life journey is as unique as their fingerprints. Different circumstances, different goals, and different timelines mean that comparing yourself to someone else is not just unhelpful, it's often misleading. Focus on your progress. Are you better than you were yesterday? That's the comparison that truly matters and the one that will help you move forward.

Embrace the fact that you will be wrong sometimes, and remain open to change and growth.

Being wrong is not a dead end; it's a detour to improvement. Just as software evolves through updates, so do we. Each version of ourselves can be better than the last, with fewer bugs and more features. When you accept that being wrong is an inevitable part of the human experience, you open yourself up to growth and learning. It's a brave step to take but a necessary one on the path to personal excellence.

Invest in Yourself

Your twenties are a goldmine of opportunity, the perfect time to invest in yourself and your education. The efforts you put into personal development during this decade can define your future. Don't hold back on investing time, effort, and yes, even money into expanding your knowledge and skill set. With the vast array of learning resources available online, the world truly is your classroom. Through your twenties, you mostly have the advantage of less risk and more time to try and explore new things. Embrace it!

People Skills Matter

You will often work in a team, and learning how to collaborate with others and communicate effectively is a crucial skill. Mastering the art of listening, explaining, leading, and discussing is essential.

In our interconnected world, social skills are just as important as technical know-how. Whether it's in the workplace or personal relationships, the ability to work in a team, communicate effectively, and empathize with others is invaluable. These skills enable you to connect, collaborate, and lead fundamental actions for success in any field.

Don't waste your time searching for "the best resource/course/book/etc."

The quest for the "best" resource can be an endless and unproductive pursuit. What's best for one may not be for another. Instead, be resourceful. Choose a resource that resonates with you and start learning. Real progress comes from application, so take what you learn and put it into practice. It's in doing that we truly understand and retain knowledge.

Life moves quickly; ensure you take time to reflect on your actions and goals.

Life moves quickly, and in the hustle, it's easy to lose sight of where we're headed. That's why it's essential to regularly pause and reflect. Are your actions still aligned with your goals? Does what you're chasing reflect what's truly important to you? These moments of reflection can recalibrate your direction, ensuring your speed is taking you where you actually want to go.

Your Life > Your Career:

Lastly, remember that your life is much more than your career. Yes, professional advancement is significant, but it's not the total sum of your existence. Your life's tapestry is woven with threads of family ties, friendships, hobbies, and passions. Make sure you're dedicating time to these aspects as well. They provide balance, happiness, and a sense of wholeness.

In Summary

These are the lessons that have guided me through my 20s. Take what resonates with you, and remember to keep learning, growing, and embracing the journey of life. Here's to the next 20 years!

This is one aspect of Python that may not be straightforward, particularly for beginners. When passing a parameter to a function, is it passed by value (a copy of the variable) or by reference (a pointer/reference to the variable)?

It depends on the variable type

We have 2 main types of variables in Python, mutable and immutable .

Mutable variables, such as lists, dictionaries, and sets, can have their content altered without the need for re-initialization.

Immutable variables, including tuples, strings, and integers, are not designed to have their content changed without re-initialization, which involves creating a new variable.

Let me show you what I mean by a simple example

>>> x = 5
>>> id(x)
4323140904
>>> x += 1
>>> id(x)
4323140936
>>> #You can see how the id of the variable x has changed
>>> l = [1]
>>> id(l)
4308117184
>>> l.append(2)
>>> id(l)
4308117184
>>> #while appending new values to the list, it still maintains the same id
>>> l = [1, 2]
>>> id(l)
4308119552
>>> # But after re-initializing it, it changes

How does that work when passing as arguments?

When you pass a variable to a function, it takes into consideration the type of this variable, and from here, it behaves accordingly

Example: Passing an immutable variable to a function

>>> x = 1
>>> def add_2(a):
...     print(f"id(a):{id(a)}")
...     a +=2
...     print(f"id(a):{id(a)}")
...     print(a)    
...
>>> id(x)
4323140776
>>> add_2(x)
id(a):4323140776
id(a):4323140840
3
>>> x
1

As you can see in the previous example, the id of the argument a was the same as x before we changed its value, but it changed after! and the value of x hasn't changed!

Example: Passing a mutable variable to a function

>>> def append_2(a):
...     print(f"id(a):{id(a)}")
...     a.append(2)
...     print(f"id(a):{id(a)}")
...     print(a)
>>> ls = [1]
>>> append_2(ls)
id(a):4308118656
id(a):4308118656
[1, 2]
>>> id(ls)
4308118656
>>> ls
[1, 2]

As you can see here, things are a little bit different!

The reference to the list didn't change even after appending a new value to the list, and the append operation reflected on the original ls list!

💡

In simpler terms, consider the main rule here is that Python passes variables by Reference. if the variable is immutable, any change you apply to it will not affect the original variable, so it will look like you are passing by value, but if it is mutable, it will keep acting like passing-by-reference as it is.

]]>

Throughout your career as a Software Engineer, working with the Shell (Command Line) is a daily routine and an essential skill. You will encounter numerous tasks that can be time-consuming, and eventually, you may discover that a specific tool could have saved you hours. Allow me to save you some trouble by providing a list of excellent tools that can transform the way you approach certain tasks.

#1 C (B)at

Yes, it's not a typo; bat is the evolution of what cat is. bat makes you able to stdout the content of a text file while keeping the formatting and text highlighting. for example, here is the difference between using bat on a Python file vs using cat

It doesn't end here; bat it offers numerous useful features that you can utilize. For instance, let's examine this file to determine whether I used spaces or tabs!

By

using the -A flag, I was able to see things like spaces, tabs, new lines ..etc, and As you can see, I used spaces inside the function and tabs inside the if statement (never do that, please!!)

bat is by default connected to git. So if you have git set up in your repo, it will read and display the changes automatically. for example,

Project homepage

#2 Grep, your search tool

grep is a very powerful utility available in UNIX and UNIX-like operating systems. grep stands for g/re/p (global/regular expression search / and print) It is used to search text or search the given file for lines containing a match to the given strings or patterns.

Basic use

grep 'pattern' filename

This will search for a 'pattern' inside the file called 'filename' and print out any lines that match.

Case Insensitive Search

grep -i 'pattern' filename

This command will match lines that contain 'Pattern', 'PATTERN', 'pattern', or any mix of cases.

Recursively Search Directories

grep -R 'pattern' directory_path

grep can be used to search directories recursively with the -r or -R option (recursive).

Search Multiple Patterns

grep -e 'pattern1' -e 'pattern2' filename

This command will print out lines that contain either 'pattern1' or 'pattern2' in the file 'filename'.

Line Number information

grep -n 'pattern' filename

With -n option, grep shows the line numbers of the lines in the output.

Count of lines with a certain pattern

grep -c 'pattern' filename

With -c option, grep will return the count of lines that match the pattern.

You can learn more about grep by exploring the options in grep --help or from the Grep Manual

#3 Crontab

Crontab is your friend when you need to schedule various jobs on your server or machine, which will be managed and executed by the cron daemon. You can use it to schedule updates and cleanups or even execute your scripts. Allow me to guide you through the main functionalities.

Listing your jobs

crontab -l

this will view the current jobs registered by the current user

Creating a new Cron

crontab -e

This will open an editor for a crontab file for you to write your corn job, you will need to use this format to describe the new corn job.

*     *     *   *    *        command to be executed
-     -     -   -    -
|     |     |   |    |
|     |     |   |    +----- day of week (0 - 6) (Sunday=0)
|     |     |   +------- month (1 - 12)
|     |     +--------- day of month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)

for example

0 0 * * 1 python3 example.py

This will execute the example.py at 12 AM every Monday

Deleting all the current registered cron jobs

crontab -r

This will remove all the current cron jobs registered for the current user. if you need access to all the system-wide cron jobs, you will need superuser privileges

Crontab Manual

#4 Xargs

Basic Usage

xargs is a powerful tool that reads and utilizes the output from one command, allowing you to apply various commands on the output with a range of options. For example:

ls | xargs

Here, xargs will get the output of ls and will print it in one line separated by spaces. As there was no command provided to xargs It will just do a print. Let's try to apply ls -lh to every output we get from ls

ls | xargs ls -lh

Combine with other tools

Mostly, you will get the most of xargs When you use it combined with other tools, for example, combine it with find

find . -type f -mtime +30 | xargs rm -f

Here, you use find to filter every file that is over 30 days old, and pass it to xargs to apply rm -f on it. you can use it combined with crontab to make it a scheduled job (it's a perfect example of using these powerful tools instead of writing a specific program for it)

Use placeholders for more complex commands

Sometimes, you need to automate commands that might need more than one argument; here, you can use the power of the placeholder. Let me give an example

ls -1 *.txt | xargs -I {} mv {} old/

Here, we define a placeholder using -I {} and use it to specify the exact location that we want our argument to be placed in.

Parallelize it

Sometimes, you would want to parallelize the job executed on the arguments supported; to do that, you will need to use -P Here is an example

find . -type f -name '*.txt' | xargs -n 1 -P 4 grep -i 'pattern'

Here, we use find to get all the text files, pass it to xargs to parallelize grep on each file using up to 4 parallel greps at once!

Xargs Manual

I come across so many handy tools every day that really make a difference in my daily work. I've shared a bunch of my favorites that I think will help you out, too. If you've got any cool tools you use in your day-to-day tasks, feel free to share them in the comments section!

Catch you in the next one!

TLDR; Avoid these 3 mistakes for clearer Python code: overusing dictionaries, using strings for variations, and not using type hinting. Consider data classes and enums for structured data and variations, and use type hinting for readability.

When you are trying to prototype an idea or start a new project in Python, it is generally a quicker process than using other languages in most cases. The simplicity of Python's syntax, side by side with its dynamic nature, is extremely attractive to anyone starting, but this can also be detrimental in the long run. As you advance in your career, you begin to see how shipping new software is just part of the big picture, and there is a larger aspect called maintaining old software, or as we call it, legacy code.

With that said, here are some mistakes to avoid if you want to make it easier for people and your future self to maintain your code.

Mistake #1 Using dictionaries for everything

Dictionaries are a fantastic data structure and incredibly powerful in Python, but overusing them can make your code difficult to trace.

The primary issue with dictionaries is their lack of a defined structure. Without a clear structure, it's challenging to know what to expect from a dictionary at a specific point in your software. What keys are present, and what types of data are held within? It will require extra effort from you to ensure the structure and the existence of certain keys/values. To avoid this, you can use Data classes.

The main advantage of Data classes is the presence of a defined structure, which can be extremely helpful if your data has a consistent structure that will always be satisfied, for example.

from dataclasses import dataclass

@dataclass
class User:
    name: str
    url: str
    age: int
    address: str
new_user = User(name="Mo", url="www.m-ashour.space", age=28, address="Earth")
#Using dataclass, you can access the data inside directly like this
>> new_user.name
>> new_user.age

#compare using dataclasses here with using dict
new_user_dict = {"name":"Mo", "url" : "www.m-ashour.space", "age":28, "address":"Earth"}
#here, you don't have an enforced defined structure and you are vurnable to more mistakes

>> new_user_dict["name"]
>> new_user_dict["age"]

As you can see, the advantage of a defined structure is quite obvious, even in a very simple example. Imagine yourself reading this code in 3 years; which one would be easier to maintain and read?

Of course, it's not always the right choice!

Data classes are easier to write and maintain, but they are not the most lightweight. if performance is an issue, you might look at other solutions, which we can discuss in another article.

Mistake #2 Using Strings to distinguish variations

Strings are powerful and incredibly useful, but misusing them can lead to numerous problems and potential issues.

To represent variations, you need a well-defined and structured method to implement them explicitly and limit the options to only the supported variations. This way, you can prevent any potential issues arising from referring to or passing a non-supported, non-implemented variation. In this case, using strings will not fulfill these requirements, but using Enums will. Allow me to use the same example.

Don't

from dataclasses import dataclass

@dataclass
class User:
    name: str
    url: str
    age: int
    address: str
    #let's add a user type
    user_type: str
#here, using user_type as string, will not give any indication on what user types we have

Please Do

from dataclasses import dataclass
from enum import Enum

class UserType(Enum):
    STANDARD = "STANDARD"
    PRO = "PRO"
    BUSINESS = "BUSINESS"

@dataclass
class User:
    name: str
    url: str
    age: int
    address: str
    user_type: UserType
#here you can clearly understand the types of users that the code supports

As you can see, Enums can provide an incredible boost in understanding the code and can provide help in how to use it. It's a timeless documentation on its own, even on the basic level. There is more to Enums, but that's a topic for a separate post.

Mistake #3 Not using type hinting

Even though type hinting in Python doesn't enforce anything or check the types during runtime (since we don't have compile time), it still provides readability on various levels.

Type hinting wasn't introduced in Python until version 3.5. While it's not native to the language or fully matured, it still offers readability and productivity enhancements when maintaining existing code or writing new code. Current Code editors and IDEs now have good integration with it, and depending on the types you hint your variables with, they can suggest appropriate methods. Additionally, it serves as basic documentation for your functions.

For example,

from typing import List, Tuple
def draw_line(x,y):
    .....

#using type hinting
def draw_line(x:List[float], y:List[float]) -> List[Tuple[float,float]]:
    .....

Can you see the difference in readability? This is a big difference for anyone reading your code (including you).

Conclusion

In this Article, I walked you through 3 main mistakes that can make your code more trouble to maintain and read. So to sum it up

• Data classes are a better option than Dicts for structured, well-defined data bundles

• Enums speak louder and clearer than Strings when it comes to domain-defined variations

• Type Hinting is a great advisor and a future guide for your colleagues and your future-self

These are not the only mistakes, there are more that we can learn from. Let me know in the comments about some of them!

See you in the next one!

]]>

Background

I have been writing Python code professionally for over half a decade (about 7 years). I have used it in various companies and products, such as Search Engines, NLP, Satellite Data Processing, Machine Learning, and more. It's a great language and has a tremendous and rich ecosystem that is hard to find in other languages, especially if you are in the data world.

That said, Python has some weaknesses that you can't overlook as you grow in your career. One of them is performance and memory management. It's not the worst (I've seen worse, believe me!), but it's not the greatest either. When you expand to scenarios where speed and memory management are crucial, Python is not the one to call.

Like most software engineers, throughout my career, I have written code in different languages, such as JS/NodeJs, TypeScript, and Golang. I was able to see how different languages somewhat translate ideas back and forth between each other. This helped me recognize what each language is good for and understand the main goals behind their creation.

Recently, I started to come across a lot of Rust content on YouTube. It's a new, interesting language that has a different mentality and design. After watching and reading more about it, I decided that I should give it a shot.

Rust Resources that I liked

The official Rust website has great resources to help you learn the language. my favorite so far is mixing between The Rust book and Rust By Example. Both work best together.

The Rust book walks you through the basics of the language and explains every bit. It contains projects and snippets so you can try it yourself and make sure to exercise.

Rust By Example takes the "learn from code and comments" approach. It doesn't go as deep as the Rust book, but it gives you a quick and easy resource to review and remember the syntax, which is quite useful.

Another Resource That I like is the Let's Get Rusty YouTube channel. It has a playlist going through the Rust Book, which is high quality and well presented!

Make no mistake, I'm not a Rust expert by any means, I'm still learning and going through the "Writing crappy Rust code" phase and fighting with the compiler to get it to accept my code.

The differences that you will experience between writing Rust and Python are not hard to notice. In fact, you can see it from the very first program you run, these are mine so far (add yours in the comment section)

Rust is a statically typed language

Everything needs to be decided at compile time; there is no room for guessing or making decisions during runtime. Which is different from how Python operates. Python is dynamically typed. Things get decided in the Runtime (when your program is running). Using type hinting in Python doesn't change this fact. It's a good practice to use, of course; I use it all the time, but it doesn't change how Python operates and doesn't enforce any kind of checks.

Rust Compiler is a strict safeguard

I sometimes use "annoying" instead of "strict," but I think I'm in a phase of my career where I appreciate it to some extent. It doesn't allow you to write code that is not safe, or that can lead to any unexpected behavior. It's a safeguard for your future self that ensures the code you write now will still work as expected no matter what time in the future. Of course, that doesn't come without a cost; the main cost here that I see is the slow write/development pace. It's hard to get it right from the beginning, and it takes time to write in Rust, especially if you are just starting. You will see the huge difference in time between writing Rust and writing Python. But I guess it gets better over time, and the correctness and safety aspects can compensate for the long development time with minimal maintenance time in the future.

No garbage collecting and no manual memory management.

Rust comes with new ideas about how to manage memory. It doesn't have a garbage collection like Python and Java, nor does it have Manual memory management like C++. It introduces the concept of Borrowing and manages it with the Borrow Checker.

The Borrow Checker in Rust enforces strict rules on how references to data can be borrowed, preventing issues like data races and dangling pointers. By keeping track of which parts of the code have access to certain data, the borrow checker allows Rust to automatically manage memory, making your code safer and more efficient.

It doesn't trust you; it knows that you will mess things up at some point, which is annoying for your ego but super helpful for your goal!

Enums are strong in Rust

I like the Enums in Rust and how it is empowered by Rust Type System. Enums in Rust let you create a type with different options, each holding different data. This makes your code shorter and clearer, helping you work with complex data and handle different situations smoothly. Enums also use Rust's strong type system and pattern matching, making your code safe and efficient.

Learning Rust made me write better Python Code

Yeah, I know it might seem silly and annoying to my own ego. But Learning Rust made me conscious about things that I didn't use to spend too much time thinking about. Writing Python makes you productive in the development time, it's super flexible and forgiving. Python trusts you most of the time that you are going to do the right thing, which is sweet but harmful. Rust doesn't give you a chance to do anything that it doesn't like, which is safe but super annoying sometimes. Learning from the strictness and the seeking for correctness from Rust and mixing it with the flexibility and simplicity of Python makes me write better code that is safer.

Conclusion

Learning different languages with different designs opens your eyes to new aspects and makes you improve. This happened to me when I learned other languages like Go, JS, TS, Java, and C, and it's happening now while learning Rust. I don't know if I'll end up proficient in Rust anytime soon (which is unlikely) or even use it in a big project. But I know that it has lots to offer, and the learning journey itself adds a lot to me.

I hope this is not the last time I write about it here, I still have lots to talk about!

It goes without saying that for most people in the tech community, when containerization is mentioned, Docker is often one of the first things that come to mind, if not the very first. I'm not here to argue that others pioneered the concept of containerization before Docker; that might be a topic for another blog post.

Instead, I want to discuss a problem I encountered - and I'm sure others have as well - while working with Docker Swarm and needing some form of data sharing between swarm nodes. Unfortunately, this feature is not natively supported in Docker. You either have to rely on a third-party storage service that offers a robust API for your nodes to interact with, which can be quite expensive, or you can take the more challenging route and create your own shared storage solution. In this post, I'll share the latter approach that I chose to implement.

Introduction:

What is GlusterFS?

What is GlusterFS?

GlusterFS is a scalable, distributed file system that combines disk storage resources from multiple servers into a single global namespace. As an open-source solution, it offers features such as replication, quotas, geo-replication, snapshots, and more.

GlusterFS allows you to consolidate multiple nodes into one namespace, presenting various options. You can:

1. Create a replicated volume that ensures the availability of your data without having to worry about data loss.

2. Establish a distributed volume that increases storage capacity by distributing your data across multiple machines. While this option sacrifices availability, it provides more storage using the same resources.

You can learn about the various setups of Gluster from here https://docs.gluster.org/en/latest/Quick-Start-Guide/Architecture/

Let's start with our setup

First, let's envision a swarm setup consisting of three manager nodes and three worker nodes. We need the containers on the worker nodes to access the same data, regardless of their location, while maintaining data consistency. The availability of our data is of utmost importance. Therefore, we need to create a replicated GlusterFS volume that duplicates all the data across multiple replication nodes. Since we lack the resources to acquire additional machines to serve as storage pools, we will utilize our swarm machines to also function as storage pool nodes.

So, our architecture will be something like this

• 3 Swarm Managers

• 3 Swarm Workers

• GlusterFs Volume connected to the 3 servers

Yes, I understand that this design might seem unusual at first, as it causes our workers to function as both storage pools and clients simultaneously. In this setup, they serve as replication nodes for the collected data while also acting as servers that mount the volume and read from it. However, upon further consideration, this configuration can be quite beneficial for a variety of use cases that your application might require, all without incurring any additional costs!

let's start the dirty work

let's assume we are working with a Debian-based distro, like Ubuntu

first, you will need to install GlusterFs on all three worker machines you can do this by

sudo apt updatesudo apt install software-properties-commonsudo add-apt-repository ppa:gluster/glusterfs-7sudo apt updatesudo apt install glusterfs-server

then, you will need to start and enable the GlusterFs daemon service on all the worker machines

sudo systemctl start glusterdsudo systemctl enable glusterd

then, make sure you generate an ssh-key on each machine

ssh-keygen -t rsa

after that, to be able to deal with all machines with their hostnames, you will need to edit the /etc/hosts for each of these machines and add the hostnames for the other nodes assigned to their IPs, like this format

<IP1> <HOSTNAME1><IP2> <HOSTNAME2>

Now, let's create our storage cluster, start from one of the machines and add the others using this command

sudo gluster peer probe <Hostname>

after you add the 2 nodes, run this command to make sure that all of the nodes joined the storage cluster

sudo gluster pool list

now, we will need to make a directory in all of the 3 workers to act like a brick for our volume But wait, what is a brick?? a brick is a directory that acts as a volume unit, and GlusterFs uses it in all of its storage pool nodes to know where is the data to store/deal with

so basically you will need to create this directory on each worker node, let's name it brick1 and put it under /gluster-volumes

sudo mkdir -p /gluster-volumes/brick1

Now, we are ready to create our replicated volume (let's call it demo-v ) [run this command only on the main machine]

sudo gluster volume create demo-v replica 3 <HOSTNAME1>:<BRICK_DIRECTORY> <HOSTNAME2>:<BRICK_DIRECTORY> <HOSTNAME3>:<BRICK_DIRECTORY> force

then start the volume

sudo gluster volume start demo-v

and Congrats, you have a replicated volume now that is ready to be mounted and used in any device Now, let's mount this volume on our 3 workers let's say that we will mount our volume under /our-application/logs

sudo mount.glusterfs localhost:/demo-v /our-application/logs/

then, to make it permanent, we will need to add it to our fstab file so open /etc/fstab and add this line

localhost:/demo-v /our-application/logs/ glusterfs defaults,_netdev,acl, backupvolfile-server=<HOSTNAME> 0 0

instead of the <HOSTNAME> add a hostname of one of the worker machines you have, so if you needed to get it out of the storage pool, you can still mount the volume using the other machine.

Now you can try this

touch /our-application/logs/test.txt

and check out the other workers, find the file? Congrats! you have a working replicated volume that is mounted across all of your workers.

In simpler words, this article shows how to use GlusterFS, a free and expandable file system, to share storage between Docker Swarm nodes. It guides you through installing and setting up GlusterFS, making a storage cluster, and testing if it works right.

Any questions? leave a comment!

]]>