The biggest problem with CI/CD pipelines

7th January 2025

Do you know the most common problem I encounter when creating or updating build or deployment pipelines?

Finding the correct path to specific files!

I think that’s why I find I have to litter my pipelines with extra steps like this:

- script: ls -alR
  displayName: "Script: List files"
  workingDirectory: $(Build.ArtifactStagingDirectory)

I think the problem is really that tools used to build software with, and the tasks you use in your pipelines, are so inconsistent with how they generate and reference files and paths.

For example:

• Does the file generated by a tool get created in a single directory, or does it create a child directory for the file to live in?
• If you use wildcards with the tool, does that change how it generates the output file(s)?
• Does the tool default to looking for files in the current working directory or somewhere else?
• If you use relative paths for tool parameters, are they relative to the current working directory, or to another specific parameter?
• If you’re creating a zip file by pointing to a directory, does it include that directory in the zip, or just the child files and directories?
• Does the tool or task support wildcards? And are they ‘normal’ wildcards (aka globbing), or are they actually regular expressions?
• If the tool supports wildcards, can you specify multiple patterns, or only one?

I’m sure there are other variations. You get the idea at least.

And once you’ve finished banging your head against the wall and got all your path ‘ducks’ in a row, do you leave all those ls -alR tasks in your pipelines just in case you might need to refer to them in the future, or do you remove them to get rid of the extra noise (and make things a tiny bit faster)?

Using a tool like Cake as an abstraction over all your tools may help to a certain extent, as it then provides a more consistent interface in how you use the tools. But even then I’ve found myself having to add extra code in my .cake files to again list what files it can see to troubleshoot when things are not working as I think they should.

It’s such a trivial thing, but it continues to trip me up, and I suspect I’m not alone 😀