Format Document
command manually, but with this extension you can format all files with a single command.
This allows you to easily ensure all files are formatted consistently.
This is especially useful if you have a large codebase and change your code formatting settings.
e.g. changing your .editorconfig
or .prettierrc
settings, or modifying your VS Code code formatting settings.
The extension still needs to manually open each file in the editor and apply the formatting, so it can take a little while to run, but it sure beats doing it manually. I hope you find this extension as helpful as I have.
Happy coding!
]]>To access the view in the Azure portal, open up the Advisor
, navigate to Workbooks
(or use this link), and open the Service Retirement (Preview)
workbook.
At the top you will see a list of services you are using that are being retired soon, the date they are being retired, how many of your resources will be affected, and a link to documentation about the retirement and actions to take.
Below that is a list of resources that will be affected by the service retirements you have selected in the list above. This tells you the resources’ subscription, resource group, name, type, location, tags, and a link to actions to take.
You can also export the results to Excel, making the information easy to share with others.
Having a single up-to-date view of all the services being retired, which resources will be affected, and links to documentation on actions that need to be taken is extremely helpful and convenient. Anybody in the organization is able to view the workbook, and as teams migrate their resources to newer services they are removed from the list, giving a realtime view of which resources still need attention.
Recommended Action: Set up a recurring reminder to check this workbook every month or quarter, so you can stay on top of any upcoming retirements that may affect you or your organization. Share this information with other teams so they can do the same.
If you are curious or just like to stay up-to-date, you can also toggle the workbook view to show all upcoming service retirements, not just retirements for the services that you are currently using. There is also the Azure Updates page that lists upcoming and past retirements, can be searched, and has an RSS feed that you can subscribe to.
Did you find this information as helpful as I did? Have any other related tips? Let me know in the comments below.
Cheers!
]]>I needed to find all of the Azure Cloud service (classic)
resources that we had in our organization, since that service is being retired on August 31, 2024.
I initially overlooked the Azure portal’s All Resources
blade, which would have allowed me to filter by resource type.
Along the way I discovered some other ways to browse and filter my Azure resources, and thought I’d share them here.
Spoiler: I show how to find the Classic services in the Azure Portal and Azure Resource Graph Explorer sections below.
Update: For an even better way to track down services that are being retired, check out my post on the Azure Service Retirement Workbook.
It is important to note that all of the approaches below require you to have access to the Azure subscription you want to explore. You can only browse and search across subscriptions you have permissions for. If you are wanting to search your entire organization, ensure you have access to all subscriptions.
The easiest and most user-friendly way to explore your Azure resources is through the Azure portal: https://portal.azure.com.
You can use the global search bar at the top center of the web page to search for resources by name, resource type (service), or resource group.
If you want to just browse your resources, navigate to the Subscriptions
blade and then drill down into the resource groups and their resources to inspect them.
NOTE: To ensure you are able to view and search all of the subscriptions you have access to, click the gear icon on the top-right of the webpage and ensure the Default subscription filter
is set to All subscriptions
, otherwise you may not see the resources you expect.
There is also an All Resources
blade that will allow you to browse, search, sort, and filter all resources in the subscriptions you have access to.
It shows you the resource Name, Type, Location, Resource Group, and Subscription, and also allows you to filter your resources by Tag.
This is a powerful blade that can be useful for finding resources of a specific type, in a specific location, or with a specific tag.
To solve my initial problem, I could use the All Resources
blade to filter by Type Cloud service (classic)
.
The portal does not always expose all of the properties of a resource, or let you search or filter by them. If you are looking for something more specific, you may need to use another approach.
Another way to browse resources is with the Azure Resource Explorer: https://resources.azure.com.
This can show you the API endpoints for your resources, and let you explore them in a tree view. It may also expose some properties that the portal does not. I have found this view is not that helpful on its own, but can be when used in conjunction with the other approaches below.
If you do not see the resources you expect, you may need to change the directory using the dropdown box at the top. If your organization has a lot of resources, the page can be very resource intensive and bring your browser to its knees.
You can leverage the Az
Azure PowerShell module to explore your resources from PowerShell.
The Az
module is actually comprised of many different modules and can be quite large.
To simply install all of the modules from the PowerShell Gallery, use the command:
Install-Module -Name Az -Repository PSGallery
However, you may prefer to only install the specific modules you need. You can view all of the various Az modules on the PowerShell Gallery.
To start, you may want to install just the Az.Resources
module, which will allow you to explore your resources:
Install-Module Az.Resources -Repository PSGallery
Once installed, use Connect-AzAccount
to connect to your Azure account.
Now that the module is installed and connected, you can use Get-AzResource
to explore the resources in the subscription you are connected to.
To list the subscriptions you have access to, use Get-AzSubscription
, and to change subscriptions, use Set-AzContext -Subscription <subscription name>
.
The PowerShell module exposes more properties than the portal does, and you can use regular PowerShell commands to filter and search for resources you want. You can also use the Az module to create, modify, and delete resources as well. For more information, check out the Az module documentation.
If you prefer to not use PowerShell, you can use the Azure CLI instead. You will need to download and install the package that is specific to your operating system.
Once you have the CLI installed, use az login
to connect to your Azure account.
Now that the CLI is installed and connected, you can use az resource list
to explore the resources in the subscription you are connected to.
To view all of the subscriptions you have access to, use az account list
, and to change subscriptions, use az account set --subscription <subscription name>
.
The CLI has subcommands for many resource types, such as az vm list
or az webapp list
to list just VMs or web apps.
Use az --help
to see all of the available commands.
The CLI may expose more properties than the portal, and allows you to also create, modify, and delete resources. For more information, check out the Azure CLI documentation.
The Azure Rest API is one of the most powerful ways to explore your resources, but it is also the most difficult.
All of the methods mentioned above use the Rest API under the hood. Using the Rest API does not allow for easy browsing of your resources like the approaches above, as you often need to know the API endpoint for the resource you are looking for. This is where the Azure Resource Explorer can help, as it can show you the API endpoints for your resources. Otherwise you often have to read through the API docs.
When testing the Rest API, it can be helpful to use a tool like Postman or Thunder Client for VS Code. There are client libraries that can make is easier to use the Rest API in your own code, and they have support for languages such as .NET, Java, Node.js, and Python.
For more information, see the Rest API documentation.
Azure Resource Graph Explorer is a service that allows you to query your Azure resources using the Kusto query language. It is extremely powerful, and you can use it from the Portal, Azure PowerShell, Azure CLI, or the Rest API.
The easiest way to use it is from the Azure portal.
From the portal, use the global search bar to search for Resource Graph Explorer
, or go to https://portal.azure.com/#view/HubsExtension/ArgQueryBlade.
You will be presented with a blank query editor, a Get started
view that has several example queries to choose from, and a tree view on the left with a search bar and the various resource types and their properties.
The example queries can show you how to use the Kusto query language, and the tree view makes it easy to add filters to your query for the resource types and properties that you are interested in.
For example, here is a Kusto query to retrieve all of the action groups (alerts) that send email notifications to me@domain.com; something that you cannot simply search for in the Action Groups or Alerts blade of the portal:
resources
| where type == "microsoft.insights/actiongroups"
| where properties['emailReceivers'][0]['emailAddress'] == "me@domain.com"
Here is a query that could be used solve my initial problem to retrieve all of the Cloud service (classic)
resources:
resources
| where type == "microsoft.classiccompute/domainnames"
| order by name desc
Here’s a screenshot of the query editor with the above query:
If needed, I could refine the query to further filter down the results based on other resource properties.
The portal allows you to download the query results as a CSV file, so they are easy to share with others. Also, unlike the PowerShell module and CLI, the Resource Graph Explorer is not scoped to a specific subscription at a time, so you can easily query across all of your subscriptions.
For more information, check out the Azure Resource Graph documentation.
So we’ve seen multiple ways you can explore your Azure resources. When should you use each one?
I want to browse my resources and see what’s there.
Use the Azure portal to navigate around your subscriptions and their resources.
I want to find a specific resource, a specific type of resource, or resources with a specific property.
Start with the Portal’s global search bar. From there, try the blade for the resource type you are looking for (e.g. the Subscriptions blade). If those do not provide the searching and filtering you need, use the Azure Resource Graph Explorer.
I want to find a specific resource or property as part of an automated script.
Use the Azure PowerShell module or Azure CLI.
I want to find resources as part of my application.
Use the Azure Rest APIs with one of the client libraries. The Azure Resource Explorer may help you find the API endpoints you need, and specific properties to filter on.
Of course, these are just recommendations, and you can use whichever approach you prefer for your scenario.
We’ve seen several ways to get started exploring your Azure resources.
For my initial problem of finding all “Cloud service (classic)” resources, I could have also used the Azure PowerShell module or Azure CLI. The Azure Portal and Azure Resource Graph Explorer can be the easiest to get started with, as they allow you to browse through services and properties easily, which is nice when you do not know the specific properties, terms, and syntax to use. Also, they do not require any additional setup (no software to install or connecting to Azure). It all comes down using the tool you are most comfortable with though.
Hopefully you’ve learned something new and this helps you find the resources you are looking for!
Happy exploring!
]]>While I didn’t realize it until a few days after recording, I was on the show’s 100th episode! Check out the episode on YouTube here. That is a huge accomplishment for the podcast, and I’m honoured to have been a part of it. Congratulations to Andrew and the rest of the team for reaching this milestone!
I was a little bummed that I did not get to also meet Jordan Hammond, as he decided to take a break from the show at the end of 2023. I missed out on meeting him by just a couple episodes. Even though he stopped just shy of 100 episodes, it is still a great feat and he should be proud. I hope he returns to the show in the future, even if only as a guest and to check off the “100 episodes milestone” box 😁.
When writing blog posts and recording videos, I have time to gather my thoughts and later update any information that I left out. While chatting live, I don’t have that luxury. After we were done recording I was soon thinking of all the things that I could have said better, or things that I left out and should have mentioned.
If I was able to edit the podcast, these are some additional things I would have mentioned:
I try to hold the content I create to a high quality standard, and am often my own worst critic. Writing these amendments shows that I still have some growth to do in learning how to relax and let go of things that are not perfect with the content I create. But, I just couldn’t help myself 😅.
Overall I think the podcast went well and I had a great time. Andrew was a great host, very easy to work with, and made me feel very comfortable, both before/after recording and while talking live. He’s a true pro! I’m looking forward to doing another show in the future.
]]>While I show several approaches here, the one I recommend using is the “Include approach with reusable workflows” (Approach 5), so you can skip straight to that section if you like and take in the sample code.
This sample GitHub repository contains all of the examples shown in this post, and you can see how the workflow runs look in the GitHub Actions web UI here. Feel free to fork the repo and play around with it yourself.
Over the summer I created the tiPS PowerShell module in GitHub and decided to use GitHub Actions for the CI/CD process. For the past few years I have been using Azure DevOps for my CI/CD pipelines, but I wanted to try out GitHub Actions to see how it compared, especially since my code was also hosted in GitHub. The approaches I show here are ones I tried out in the tiPS project as it evolved, until settling on an approach I was happy with.
Azure DevOps and much of the industry use the term “pipeline” to refer to the automated steps to build and deploy software, but GitHub Actions uses the term “workflow” instead. For the purposes of this post, the terms “pipeline” and “workflow” are interchangeable.
Similarly, Azure DevOps uses the term “template”, while GitHub Actions uses the term “reusable workflow”, so I may use them interchangeably as well.
Some CI/CD criteria we want to meet in our examples are:
staging
environment automatically when a main
branch build completes successfully.production
environment.
e.g. A manual approval.The approaches we will look at are:
pull
approach).push
approach).include
approach).include
approach with reusable workflows).I typically prefer to use the include
approach, but I’ll show each approach so you can decide which one you prefer for a given scenario.
If you are curious or confused about any of the workflow yaml syntax shown in the examples below, checkout the workflow syntax for GitHub Actions docs.
I created this sample GitHub repository that contains all of the examples shown in this post, so you view their code and can see how they look in the GitHub Actions web UI.
In the example yaml code below, I use the “👇” emoji to call out specific things to take note of, or that have changed from one approach to the next.
Here is an example of a single workflow file that builds and deploys some code:
name: 1-single-file--build-and-deploy
on:
pull_request:
branches: main # Run workflow on PRs to the main branch.
# Run workflow on pushes to any branch.
push:
# Allows you to run this workflow manually from the Actions tab.
workflow_dispatch:
env:
artifactName: buildArtifact
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout the repo source code
uses: actions/checkout@v3
# Steps to version, build, and test the code go here.
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./ # Put the path to the build artifact files directory here.
deploy-to-staging:
# 👇 Only run this deploy job after the build-and-test job completes successfully.
needs: build-and-test
runs-on: ubuntu-latest
# 👇 Only run on pushes (not PRs) or manual triggers to the main branch.
if: (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/main'
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./buildArtifact
# Steps to deploy the code go here.
deploy-to-production:
# 👇 Only run this deploy job after the deploy-to-staging job completes successfully.
needs: deploy-to-staging
runs-on: ubuntu-latest
environment: production # Used for environment-specific variables, secrets, and approvals.
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./buildArtifact
# Steps to deploy the code go here.
There are a few things to note here.
First, the workflow is automatically triggered when a PR to the main branch is created, or when a change is pushed to any branch.
It can also be manually triggered.
Second, the workflow has 3 jobs: build-and-test
, deploy-to-staging
, and deploy-to-production
.
Notice in the deploy-to-staging
job we use a conditional if
statement to ensure we do not deploy if the workflow was triggered by a PR, or if the push
was not for the main
branch.
Technically we did not need to create separate deploy jobs, and could have just put the deployment steps in the build-and-test
job.
In general, it is a good idea to separate the build steps from the deployment steps to maintain a separation of concerns.
A technical reason for keeping them separate is GitHub Actions allows you to add approvals to a job via the environment
key.
Approvals are often used to block deployments until someone manually approves it.
Only jobs support an environment
, and you would typically have a deployment job for each environment that you need to deploy to.
Lastly, if you do decide to use a single job for both the build and deployment steps, then you technically do not need the Upload artifact
and Download artifact
steps.
I would still recommend using the Upload artifact
step though so that the build artifact is available for download in the GitHub Actions UI, in case you need to inspect its files.
Pros:
env.artifactName
Cons:
main
.The deployment jobs/steps will be skipped for PRs and branch builds, but will still show up in the workflow web UI. This can be confusing to users, as they may not understand why those jobs/steps were skipped and that they cannot be manually triggered.
The PR builds and non-main branch builds will show up in the same 1-single-file--build-and-deploy
workflow runs as the main
branch builds.
If there are a lot of PR runs or pushes to branches, they may bury the main
branch runs, forcing you to go back several pages in the web UI to find the main
branch runs to answer questions like, “When was the last time we deployed to production?”.
)
To see what the GitHub Actions UI looks like with this approach, check out the workflow runs in the sample repository.
Here is an example of a workflow file that just builds the code:
name: 2-pull--build
on:
pull_request:
branches: main # Run workflow on PRs to the main branch.
# Run workflow on pushes to any branch.
push:
# Allows you to run this workflow manually from the Actions tab.
workflow_dispatch:
env:
artifactName: buildArtifact
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout the repo source code
uses: actions/checkout@v3
# Steps to version, build, and test the code go here.
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./ # Put the path to the build artifact files directory here.
And the accompanying workflow file that deploys the code using the pull approach:
name: 2-pull--deploy
on:
# 👇 Run workflow anytime the 2-pull--build workflow completes for the main branch.
# Unfortunately, can not have it only run on successful builds, so it will run when builds fail too.
workflow_run:
workflows: 2-pull--build
types: completed
branches: main
# Allows you to run this workflow manually from the Actions tab.
workflow_dispatch:
inputs:
# 👇 Must specify the build artifacts to deploy when running manually.
workflowRunId:
description: 'The build workflow run ID containing the artifacts to use. The run ID can be found in the URL of the build workflow run.'
type: number
required: true
env:
artifactName: buildArtifact # This must match the artifact name in the 2-pull--build workflow.
# 👇 Ternary operator to use input value if manually triggered, otherwise use the workflow_run.id of the workflow run that triggered this one.
workflowRunId: ${{ github.event_name == 'workflow_dispatch' && inputs.workflowRunId || github.event.workflow_run.id }}
jobs:
deploy-to-staging:
# 👇 Only run the deployment if manually triggered, or the build workflow that triggered this succeeded.
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
steps:
# 👇 Must use a 3rd party action to download artifacts from other workflows.
- name: Download artifact from triggered workflow
uses: dawidd6/action-download-artifact@v2
with:
run_id: ${{ env.workflowRunId }}
name: ${{ env.artifactName}}
path: ./buildArtifact
search_artifacts: true
# Steps to deploy the code go here.
deploy-to-production:
# Only run this deploy job after the deploy-to-staging job completes successfully.
needs: deploy-to-staging
runs-on: ubuntu-latest
environment: production # Used for environment-specific variables, secrets, and approvals.
steps:
# Must use a 3rd party action to download artifacts from other workflows.
- name: Download artifact from triggered workflow
uses: dawidd6/action-download-artifact@v2
with:
run_id: ${{ env.workflowRunId }}
name: ${{ env.artifactName}}
path: ./buildArtifact
search_artifacts: true
# Steps to deploy the code go here.
Here the build workflow is separate from the deployment workflow.
The build workflow is triggered when there is a push to any branch, a PR to the main branch, or when manually triggered.
The deployment workflow uses the on.workflow_run
trigger to wait and listen for the build workflow to complete against the main branch.
Because the build uses its own workflow, the deployment workflow needs a reference to the build’s workflow run ID so it knows which build run to download the artifacts from.
This is provided automatically when the build triggers the deployment workflow, but must be provided manually when the deployment workflow is manually triggered.
You can find the build workflow run ID by opening the build workflow run in the GitHub Actions UI and looking at the URL.
The URL will look something like https://github.com/deadlydog/GitHub.Experiment.CiCdApproachesWithGitHubActions/actions/runs/6985605790
, where the run ID is 6985605790
.
The next thing to note is that the artifactName
environment variable is duplicated in both the build and deployment workflows.
We could have the build workflow create an output variable that the deployment workflow could reference, but for the sake of simplicity I just duplicated the environment variable here.
Next, notice that the deploy-to-staging
job has a conditional if
statement that will only run the job if the workflow was manually triggered, or if the build workflow completed successfully.
Unfortunately, at this time, the on.workflow_run
event does not have a property to indicate that the deploy workflow should only be triggered if the build workflow completed successfully, so we have to do the check ourselves on the job.
Lastly, the deployment jobs use a 3rd party action to download the build artifact. At this time, GitHub Actions does not have a built-in action to download artifacts from other workflows. They do provide API endpoints to download artifacts from other workflows, but it is simpler to use the 3rd party action.
Pros:
The build workflow only shows the build steps in the GitHub Actions UI, and the deployment workflow only shows the deployment steps, so we do not constantly have “skipped” jobs for non-deployment builds like in the previous single-workflow-file approach.
Cons:
The deployment workflow is triggered even if the build workflow for the main branch fails, resulting in skipped deployment runs showing in the workflow UI. This can be confusing to users, and it clutters the workflow UI.
The name of the deployment workflow run is always 2-pull--deploy
, rather than the commit message of the build workflow run that triggered it.
It also does not show the commit SHA.
This can make it difficult to find the deployment workflow run you are looking for in the GitHub UI.
Certain variables must be duplicated between the build and deployment workflows (e.g. env.artifactName
), or additional code added to pass the variables between the workflows.
To see what the GitHub Actions UI looks like with this approach, check out the workflows in the sample repository for the pull build runs and pull deploy runs.
After using Azure DevOps classic pipelines for years, this approach felt very natural and is probably the one most similar to classic pipelines. In Azure DevOps, you would explicitly create separate build and deployment pipelines, and the first step of the deployment pipeline setup is specifying the build pipeline that it should pull the artifacts from, and potentially automatically trigger off of. So the build pipeline would not know anything about the deployment pipeline, and the deployment pipeline could be automatically triggered when the build pipeline completed successfully.
This approach worked quite well in GitHub at first, but I really did not like how blank, skipped deployment workflow runs got created when the main branch build failed. It quickly cluttered up the deployment runs when issues were encountered with the build workflow that took many attempts to fix. Also, having every deployment run named the same thing made finding a specific deployment very painful.
Rather than triggering the deployment workflow when a build workflow completes, you may want to trigger the deployment workflow on other events, such as when a new image is uploaded to a container registry, or when a new release is created. These are valid scenarios that may be suitable for your project. One thing to consider is that you will either need to no longer upload the artifact to the container registry for non-main branch builds, meaning you are not able to inspect and test them, or you will need to add code to your deployment workflow to determine if the artifact should be deployed or not. This may or may not be a big deal, depending on your project requirements.
Here is an example of a workflow that builds the code and then triggers a deployment workflow:
name: 3-push--build
on:
pull_request:
branches: main # Run workflow on PRs to the main branch.
# Run workflow on pushes to any branch.
push:
# Allows you to run this workflow manually from the Actions tab.
workflow_dispatch:
inputs:
# 👇 Allow deploying non-main branch builds.
deploy:
description: 'Deploy the build artifacts. Only has effect when not building the main branch.'
required: false
type: boolean
default: false
env:
artifactName: buildArtifact
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout the repo source code
uses: actions/checkout@v3
# Steps to version, build, and test the code go here.
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./ # Put the path to the build artifact files directory here.
# 👇 Trigger the deployment workflow.
trigger-deployment:
needs: build-and-test
# 👇 Only trigger a deployment if the deploy parameter was set, or this build is for a push (not a PR) on the default branch (main).
if: inputs.deploy || (github.event_name != 'pull_request' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch))
uses: ./.github/workflows/3-push--deploy.yml
# 👇 Allow the deployment workflow to access the secrets of this workflow.
secrets: inherit
And here is the accompanying deployment workflow:
name: 3-push--deploy
on:
# 👇 Allow this workflow to be called by the 3-push--build workflow.
workflow_call:
env:
artifactName: buildArtifact # This must match the artifact name in the 3-push--build workflow.
jobs:
deploy-to-staging:
runs-on: ubuntu-latest
steps:
# 👇 Can use the native download-artifact action.
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./buildArtifact
# Steps to deploy the code go here.
deploy-to-production:
# Only run this deploy job after the deploy-to-staging job completes successfully.
needs: deploy-to-staging
runs-on: ubuntu-latest
environment: production # Used for environment-specific variables, secrets, and approvals.
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./buildArtifact
# Steps to deploy the code go here.
Once again the build workflow is separate from the deployment workflow. The build workflow will trigger on a push to any branch, PRs to the main branch, or when manually triggered. Rather than the deploy workflow listening for the build workflow to complete, the build workflow explicitly calls the deploy workflow as its final job.
Looking at the trigger-deployment
job, we can see that only builds made from the main branch will trigger the deployment workflow.
A deploy
parameter is also provided in the build workflow that can be set when manually triggering a build, allowing for non-main branch builds to be deployed as well, if needed.
Notice that the job provides the secrets: inherit
key, which allows the deployment workflow to access the secrets of the build workflow.
Without this, the deployment workflow would not have access to the GitHub repository secrets.
Aside: In addition to passing secrets to the deployment workflow, you can also pass other parameters to the deployment workflow by using the
with
key. While none are shown in this example, I will mention that in order to pass non-string values (e.g. boolean or number), I had to use thefromJSON
function to maintain the variable’s type, as shown in this GitHub issue.
Looking at the deployment workflow, you can see we are using the on: workflow_call
event to allow the workflow to be called by the build workflow.
Since the build workflow is triggering the deployment workflow, the end result is a single workflow run, meaning we can use the native actions/download-artifact
action to download the build artifact, rather than having to use a 3rd party action.
Pros:
Cons:
Since the deployment workflow is never triggered by GitHub, but is instead called by the build workflow, it means the deployment workflow will never show any runs.
Instead, the deploy jobs will show up as part of the build workflow run.
This means that builds for PRs and non-main branches will be mixed in with the main
branch builds and deployments.
Just like with the 1-single-file--build-and-deploy
approach, this may bury the deployments under several pages of non-main branch runs, making it difficult to find runs you care about in the GitHub UI.
Since the deployment jobs show up in the build workflow run, the GitHub UI prefixes each of the deployment jobs with the name of the job from the build workflow. This can make it difficult to see the full name of the deployment jobs in the generated diagram. Thankfully, the jobs are listed on the left in a tree view as well, so you can still read the full name there more easily.
To see what the GitHub Actions UI looks like with this approach, check out the workflows in the sample repository for the push build runs and push deploy runs.
Here is an example of a workflow that builds the code:
name: 4-include--build
on:
pull_request:
branches: main # Run workflow on PRs to the main branch.
# 👇 Run workflow on pushes to any branch, except the main branch.
push:
branches-ignore: main
# Allows you to run this workflow manually from the Actions tab.
workflow_dispatch:
# 👇 Allows this workflow to be called from the deployment workflow.
workflow_call:
env:
artifactName: buildArtifact
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout the repo source code
uses: actions/checkout@v3
# Steps to version, build, and test the code go here.
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./ # Put the path to the build artifact files directory here.
And here is the accompanying deployment workflow:
name: 4-include--deploy
on:
# 👇 Trigger the workflow on a push to the main branch.
push:
branches: main
# 👇 Allows you to run this workflow manually (for any branch) from the Actions tab.
workflow_dispatch:
env:
artifactName: buildArtifact # This must match the artifact name in the 4-include--build workflow.
jobs:
# 👇 Call the build workflow to create the artifacts to deploy.
build-and-test:
uses: ./.github/workflows/4-include--build.yml
secrets: inherit # Pass secrets to the build workflow, if necessary.
deploy-to-staging:
# 👇 Only run this deploy job after the build-and-test job completes successfully.
needs: build-and-test
runs-on: ubuntu-latest
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./buildArtifact
# Steps to deploy the code go here.
deploy-to-production:
# Only run this deploy job after the deploy-to-staging job completes successfully.
needs: deploy-to-staging
runs-on: ubuntu-latest
environment: production # Used for environment-specific variables, secrets, and approvals.
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./buildArtifact
# Steps to deploy the code go here.
You will notice that we still have separate build and deployment workflows. One key difference here is the specific triggers for each workflow. They have been set up so that the build workflow is only triggered by non-deployment builds, and the deployment workflow is triggered by builds that are meant to be deployed.
The build workflow will trigger on a push to any branch EXCEPT the main branch, PRs to the main branch, or when manually triggered.
It also allows other workflows to call it via the workflow_call
event.
The build workflow no longer triggers deployments, neither directly (push) nor indirectly (pull).
The deployment workflow will now trigger on a push to the main branch, or when manually triggered.
The manual trigger allows non-main branch deployments if needed.
A key thing to note here is that the deployment workflow includes the build workflow via the uses
key, so when a deployment is triggered it will first run the build jobs as part of its workflow run.
This is similar to the push
approach mentioned earlier, except that the dependency has been inverted so instead of the build workflow calling the deployment workflow, the deployment workflow calls the build workflow.
This improves the workflow UI experience, as the deployment jobs will show up as part of the deployment workflow run, rather than the build workflow run like they did with the push approach.
I came across this approach on this excellent blog post when I wasn’t satisfied with the pull and push approaches.
Pros:
Cons:
In the deployment workflow run GitHub UI, the build job name is prefixed with the name of the build job in the deployment workflow, which is a minor annoyance.
To see what the GitHub Actions UI looks like with this approach, check out the workflows in the sample repository for the include build runs and include deploy runs.
You probably noticed that we are always deploying to 2 environments: staging and production. You may have even more environments that you need to deploy to. This results in a lot of duplicate code in the deployment workflows.
To avoid the duplicate code, we can use a reusable workflow to define the deployment jobs, and then include the reusable workflow in the deployment workflow. Azure DevOps calls these “templates”, but GitHub Actions calls them “reusable workflows”. You can think of a reusable workflow as a function that accepts parameters, so you define it once, and then can call it multiple times with different parameters.
One caveat to be aware of is that while templates may also include other templates, GitHub only allows up to 4 levels of template nesting.
Also, a workflow may only call up to 20 other workflows, including nested ones.
And just like regular workflows, reusable workflows must be placed directly in the .github/workflows
directory.
See the GitHub docs on reusable workflows for more information and limitations.
Although I am only now introducing reusable workflows here, we’ve actually already been using them in the push
and include
approaches above, but were not calling them multiple times.
Let’s see how to do that now.
Here is an example of a workflow that builds the code:
name: 5-include-with-deploy-template--build
on:
pull_request:
branches: main # Run workflow on PRs to the main branch.
# Run workflow on pushes to any branch, except the main branch.
push:
branches-ignore: main
# Allows you to run this workflow manually from the Actions tab.
workflow_dispatch:
# 👇 Allows this workflow to be called from the deployment workflow, but the parameters must be provided.
workflow_call:
inputs:
artifactName:
description: The name of the artifact to upload to.
required: true
type: string
env:
# 👇 Provide a default artifact name for when this workflow is not called by the deployment workflow.
artifactName: ${{ inputs.artifactName || 'buildArtifact' }}
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout the repo source code
uses: actions/checkout@v3
# Steps to version, build, and test the code go here.
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: ${{ env.artifactName }}
path: ./ # Put the path to the build artifact files directory here.
And here is the accompanying deployment workflow:
name: 5-include-with-deploy-template--deploy
on:
# Trigger the workflow on a push to the main branch.
push:
branches: main
# Allows you to run this workflow manually (for any branch) from the Actions tab.
workflow_dispatch:
env:
# 👇 Set the artifact name that will be used by the build and deployments, so it is now only defined in one place.
artifactName: buildArtifact
jobs:
# 👇 Call the build workflow to create the artifacts to deploy, and provide the artifact name.
build-and-test:
uses: ./.github/workflows/5-include-with-deploy-template--build.yml
with:
artifactName: ${{ github.env.artifactName }}
secrets: inherit # Pass secrets to the build workflow, if necessary.
deploy-to-staging:
# Only run this deploy job after the build-and-test job completes successfully.
needs: build-and-test
# 👇 Call the deploy template with the proper environment name to deploy the artifacts.
uses: ./.github/workflows/5-include-with-deploy-template--deploy-template.yml
with:
artifactName: ${{ github.env.artifactName }}
environmentName: staging
secrets: inherit # Pass repository secrets to the deployment workflow.
deploy-to-production:
# Only run this deploy job after the deploy-to-staging job completes successfully.
needs: deploy-to-staging
# 👇 Call the deploy template with the proper environment name to deploy the artifacts.
uses: ./.github/workflows/5-include-with-deploy-template--deploy-template.yml
with:
artifactName: ${{ github.env.artifactName }}
environmentName: production
secrets: inherit # Pass repository secrets to the deployment workflow.
We now have on additional workflow file, which is the reusable workflow (template) that defines the deployment job:
name: 5-include-with-deploy-template--deploy-template
on:
# 👇 Allows this workflow to be called from the deployment workflow, but the parameters must be provided.
workflow_call:
inputs:
artifactName:
description: The name of the artifact to download and deploy.
required: true
type: string
environmentName:
description: The name of the environment to deploy to.
required: true
type: string
jobs:
deploy:
runs-on: ubuntu-latest
# 👇 Allows using variables and secrets defined in the provided environment.
environment: ${{ inputs.environmentName }}
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: ${{ inputs.artifactName }}
path: ./buildArtifact
# Steps to deploy the code go here.
I could have left the build workflow identical to the include
approach shown earlier, however I thought that I would show how to allow the artifact name to be provided as a parameter.
This allows the deployment workflow to provide the artifact name to the build workflow, so that we do not have to hardcode the value in 2 separate files as we had been doing earlier.
Since the build may still be triggered outside of a deployment workflow, we had to update the env: artifactName
to use a default value when the input value is not provided.
This same approach can be used for other values that you want shared between different workflows.
This excellent blog post shows how to also to use the ternary operator to provide a default value, and also explains why we cannot use env
variables in a job’s if
clause.
Next we see the deployment workflow.
As mentioned above, you can see that we now pass the artifact name to the build workflow, so the only place the artifact name value is defined is in the deployment workflow.
Next, notice that the workflow no longer duplicates the deployment steps in the deploy-to-staging
and deploy-to-production
jobs, but instead calls the new reusable workflow with the appropriate parameters; namely the environmentName
.
In our example the deployment code was simply # Steps to deploy the code go here.
, but in a real world scenario the deployment steps may be several hundred lines of yaml code, so not duplicating it is a big win.
Finally, look at the reusable workflow.
You can see it takes 2 parameters: artifactName
and environmentName
.
It defines a single job that is used to perform the deployment.
GitHub allows you to create environment-specific variables and secrets, which can be used by the jobs in the workflow.
These are configured in the GitHub repository web UI under the “Settings” tab, and then the “Environments” menu item.
I personally prefer to have the variable values defined directly in the workflow files so that they are under source control.
If you take that approach, you would simply add additional parameters to the reusable workflow and pass them in from the deployment workflow, just like we did with the artifactName
and environmentName
parameters.
Secrets of course should not be committed to source control, so you would still want to define those in the GitHub UI, or use a 3rd party secret manager like Azure Key Vault.
This approach has the same pros and cons as the include
approach above, as well as:
Pros:
Cons:
Similar to 3-push--deploy
above, because the template will only be referenced by other workflows and never directly triggered itself, it still shows up in the GitHub Actions UI as a workflow, but will never have any runs shown, which may be confusing to users.
To see what the GitHub Actions UI looks like with this approach, check out the workflows in the sample repository for the build runs and deploy runs.
GitHub does not allow you to place reusable workflows in subdirectories, so they are mixed in with other workflow files.
It is a good idea is to prefix them with a word like template
to make it clear that they are reusable workflows, and not meant to be triggered directly.
When using the include
approach and this one, you probably do not want to name your workflow just deploy
, since it actually both builds and deploys.
You may want to name the workflows something like build-and-test
and build-test-and-deploy
to more accurately reflect what they do, and call the deployment reusable workflow something like template-deploy
.
This is the approach I typically use for my projects.
In this post we’ve seen a number of different approach you can take to define your build and deploy workflows.
I typically prefer the last include with reusable workflows
approach, but you may prefer another depending on your needs.
The example code I’ve shown here is very simple and should simply be used as a starting point. You may want additional triggers for your workflows, such as when a tag is created, or on a schedule. You may want to create additional templates for other operations, such as running load tests, or running a security scan.
My goal with this post was to share the various ways I’ve found to structure GitHub CI/CD workflows, and hopefully give you some ideas on how to structure your own.
Finally, a reminder that I also created this sample GitHub repository that contains all of the examples shown in this post, so you view the code and can see how they look in the GitHub Actions menu.
Have another approach that you think should be on this list, or other relevant information? Leave a comment and let me know.
Happy pipeline building!
]]>First
, Single
, and SingleOrDefault
methods.
In this post we’ll look at why these methods can be harmful, and what you should do instead.
Single
, SingleOrDefault
, and First
throw exceptions that are vague and unhelpful.
You are better off to write the logic yourself and include rich error/log information that will help with troubleshooting.
Single
, SingleOrDefault
, First
, and FirstOrDefault
are LINQ extension methods that return the first element of a sequence that satisfies a specified condition.
The typical use cases for these methods are:
Single
to retrieve an item, ensuring that one and only one item matches the condition.SingleOrDefault
to retrieve an item, and ensure that it is the only one that matches the condition.
If no items match the condition a default item is returned.First
to retrieve the first item, and throw an exception if no matches are found.
It does not care if multiple items match the condition.FirstOrDefault
to retrieve the first item.
If no items match the condition a default item is returned, and it does not care if multiple items match the condition.Here is an example of using them in C#:
var people = new List<string> { "Alfred Archer", "Billy Baller", "Billy Bob", "Cathy Carter" };
---
var alfred = people.First(x => x.StartsWith("Alfred"));
var billy = people.First(x => x.StartsWith("Billy")); // Returns Billy Baller
var zane = people.First(x => x.StartsWith("Zane")); // Throws an exception
---
var alfred = people.FirstOrDefault(x => x.StartsWith("Alfred"));
var billy = people.FirstOrDefault(x => x.StartsWith("Billy")); // Returns Billy Baller
var zane = people.FirstOrDefault(x => x.StartsWith("Zane")); // Returns null
---
var alfred = people.Single(x => x.StartsWith("Alfred"));
var billy = people.Single(x => x.StartsWith("Billy")); // Throws an exception
var zane = people.Single(x => x.StartsWith("Zane")); // Throws an exception
---
var alfred = people.SingleOrDefault(x => x.StartsWith("Alfred"));
var billy = people.SingleOrDefault(x => x.StartsWith("Billy")); // Throws an exception
var zane = people.SingleOrDefault(x => x.StartsWith("Zane")); // Returns null
When First
and Single
do not find any items that match the condition, they throw the following exception:
System.InvalidOperationException: Sequence contains no matching element
When Single
and SingleOrDefault
find more than one element that satisfies the condition, they throw the following exception:
System.InvalidOperationException: Sequence contains more than one matching element
Both of these exception messages are vague and unhelpful. Imagine seeing it in your application logs, or worse, returned to the user. Would you know right away what the problem was and how to fix it? Even with a stack trace, which we do not always have, it may not be obvious.
The message doesn’t tell us what dataset was searched, what the condition was, or what elements satisfied the condition. This is crucial information to know when troubleshooting what went wrong.
Not only is the exception message unhelpful, but throwing an exception is an expensive operation that is best to avoid when possible.
FirstOrDefault
is the only method that does not throw an exception, and thus is the only one that I would recommend using.
Simply using FirstOrDefault
instead of SingleOrDefault
is not a good solution.
If you were considering using SingleOrDefault
, then you are probably trying to validate that only a single item was returned, as more than one item returned may mean that something is wrong with your query, your data, your code, or your business logic.
If we just use FirstOrDefault
instead, it hides the issue that multiple items matched the search criteria and we may never realize that there is a problem.
In our example above, when using FirstOrDefault
the billy
variable would be set to “Billy Baller” and there would be no indication that there was another “Billy” in the dataset.
This could lead to problems if our logic expects there to only be one “Billy” in the dataset and makes decisions based on that assumption.
Similarly for First
and Single
, simply using FirstOrDefault
in their place without additional validation to ensure that a result was found is a recipe for disaster.
One solution is to use Where
instead of Single
and SingleOrDefault
, and then explicitly validate that only a single item was returned.
This allows us to return a rich error message, as well as avoid the expensive exception being thrown if we want.
Here is one example of how we could do this:
var people = new List<string> { "Alfred Archer", "Billy Baller", "Billy Bob", "Cathy Carter" };
var name = "Billy";
var persons = people.Where(x => x.StartsWith(name)).ToList();
if (persons.Count > 1)
{
throw new TooManyPeopleFoundException(
$"Expected to only find one person named '{name}', but found {persons.Count} in our people list: {string.Join(", ", persons)}");
}
You can see that the exception thrown contains much more information to help troubleshoot the problem, such as:
TooManyPeopleFoundException
.If we also wanted to ensure that at least one item was found, we could add the following check:
if (persons.Count == 0)
{
throw new PersonNotFoundException($"Could not find a person named '{name}' in our people list.");
}
This is just one example of how you could implement this. You might choose to create your own SingleItemOrDefault helper method or extension method that performs the operations and adds the information to the exception. You might not want to throw an exception at all, but instead use the Result pattern to return a failed result with the rich information.
The above shows how to avoid using Single
and SingleOrDefault
.
Let’s see how we can avoid using First
as well.
Rather than using First
we can leverage FirstOrDefault
.
FirstOrDefault
is preferred over Where(...).ToList()
and checking Count == 0
to avoid iterating over the entire collection when a match exists.
Here’s an example of using FirstOrDefault
instead of First
:
var name = "Zane";
var person = people.FirstOrDefault(x => x.StartsWith(name));
if (person == null)
{
throw new PersonNotFoundException($"Could not find a person named '{name}' in our people list.");
}
You will need to be mindful of the default value for the type you are working with. Object types will be null, but value types will be their default value.
The important thing is that you do not rely on the default InvalidOperationException
, and that your error message includes all the information that will help you troubleshoot any issues.
This is a good general rule to follow for any error logging you perform.
Depending on the sensitivity of the data, you may need to be careful about which information you include in the error.
While you could do something like this:
var name = "Billy";
try
{
var person = people.SingleOrDefault(x => x.StartsWith(name));
}
catch (InvalidOperationException ex)
{
var persons = people.Where(x => x.StartsWith(name)).ToList();
throw new TooManyPeopleFoundException(
$"Expected to only find one person named '{name}', but found {persons.Count} in our people list: {string.Join(", ", persons)}", ex);
}
This is not performant.
We’ve already mentioned that throwing exceptions is expensive, and this code now throws two.
More importantly, the SingleOrDefault
call enumerates over the entire collection once.
In order to get the useful information to include in the error message, it has to enumerate the entire collection again using the Where
query, so why not just do that in the first place so we only traverse the collection once?
You might look at the recommended code and wonder if the extra few lines of code are worth it. I guarantee you it is. You can even write a helper or extension method to make the pattern easier to use.
Spending an extra few minutes to add detailed information to your errors will save you and your team hours of troubleshooting in the future. There are some scenarios where it is impossible to ever solve the root issue without this extra information, such as when validating ephemeral data that no longer exists by the time you get to troubleshooting.
Over the past decade I have seen people advocate for First
, Single
, and SingleOrDefault
.
In theory they are a good idea, but the current .NET implementation leads to more problems than it is worth.
Until the method is updated to at least allow you to provide a custom error message that includes extra information, I always caution against using them and instead recommend writing the logic yourself.
I even went so far as to create a checkin policy that would prevent developers from committing code that used First
, Single
, and SingleOrDefault
in our flagship product.
That should give you an idea of how many developer and support staff hours were wasted tracking down “Sequence contains no matching element” and “Sequence contains more than one matching element” errors.
Similar logic could be implemented today as a Roslyn analyzer.
I hope you’ve found this post helpful, and that it saves future you countless hours of troubleshooting.
Happy coding!
]]>While testing my tiPS PowerShell module with Pester, I ran into a scenario where I wanted to mock out a function which returned a path to a configuration file.
Pester has a built-in PSDrive (see MS docs) called TestDrive
, which is a temporary drive that is created for the duration of the test (see the docs here) and automatically cleaned up when the tests complete.
So rather than hardcoding a local path on disk, you can use a path like TestDrive:/config.json
.
The problem was that the .NET methods, like System.IO.File.WriteAllText()
and System.IO.File.ReadAllText()
, do not work with the TestDrive:
PSDrive, as they are unable to resolve the path.
The Set-Content
and Get-Content
cmdlets work fine with TestDrive:
, but I wanted to use the .NET methods for performance reasons.
I thought an easy fix would be to just use Resolve-Path
or Join-Path -Resolve
to get the full path, but they return an error when the file or directory does not exist.
I did not want to manually create the file in my mock or my test, as I wanted my Pester test to ensure the module created the file properly.
This is when I stumbled upon the GetUnresolvedProviderPathFromPSPath
method, which can be used to resolve a file or directory path, even if it does not exist.
Here is an example of how to use it:
[string] $configPath = $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath('TestDrive:/config.json')
This resolved the path to the Windows temp directory on the C drive, and I was able to use it with the .NET System.IO.File
methods.
This method works with any path, not just PSDrive paths. For example, it also resolves this non-existent path:
$ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath("$Env:Temp/FileThatDoesNotExist.txt")
to C:\Users\DAN~1.SCH\AppData\Local\Temp\FileThatDoesNotExist.txt
.
Later I fully read the Pester TestDrive documentation and found that it actually has a built-in $TestDrive
variable that is compatible with the .NET methods 🤦♂️.
So instead of using TestDrive:/config.json
, I could just use $TestDrive/config.json
.
I ended up changing my Pester mock to use this instead, as it is much cleaner:
[string] $configPath = "$TestDrive/config.json"
Oh well, I learned something new about PowerShell, so it was worth it.
I’m sure I’ll run into another situation down the road where GetUnresolvedProviderPathFromPSPath
will come in handy.
Hopefully you’ve learned something new too.
Happy coding!
]]>class Person {
[string] $Name
[int] $Age
}
This blog post is a great overview of PowerShell classes.
Add-Type -Language CSharp -TypeDefinition @'
public class Person {
public string Name { get; set; }
public int Age { get; set; }
}
'@
When using this method, to make editing easier and still get syntax highlighting and intellisense, I recommend putting the C# code in a .cs file and importing it with:
[string] $csharpCode = Get-Content -Path "C:\Path\To\Person.cs" -Raw
Add-Type -Language CSharp -TypeDefinition $csharpCode
public class Person {
public string Name { get; set; }
public int Age { get; set; }
}
The C# code is then compiled using Visual Studio / MSBuild / dotnet.exe to create a dll file, and the dll is imported in PowerShell with:
Add-Type -Path "C:\Path\To\Person.dll"
I created this PowerShell.Experiment.ClassPerformanceComparison repository to test and compare the 3 different ways to define and import classes and enums. The test defines identical classes that are imported using each method, and the time required to import each class is measured. It defines a very basic class, and a slightly larger and more complex class, and duplicates each class 3 times just with a different name, so there are 6 classes in total that are imported. For more details, see the repo and the code.
The results of the test are below:
Class | PowerShell Classes | Inline C# Classes | Compiled Assembly C# Classes |
---|---|---|---|
Basic1 | 89ms | 764ms | 10ms |
Basic2 | 10ms | 25ms | 9ms |
Basic3 | 9ms | 36ms | 8ms |
Large1 | 13ms | 230ms | 8ms |
Large2 | 11ms | 55ms | 8ms |
Large3 | 12ms | 28ms | 8ms |
The first PowerShell class declared is a little slow to import, but subsequent declarations are much faster. I suspect this is due to PowerShell loading some assemblies into memory that it needs to import a PowerShell class/enum. Once those assemblies are loaded, subsequent imports are much faster.
It appears that the size of the class may have a bit of impact on the import time, but it is not significant.
Inline C# classes are very slow to import. I suspect this is because the C# code is compiled at runtime before being loaded into memory. The first import is especially slow. Also, any dependencies that the class references need to be loaded into memory. The Large class references some types not referenced by the Basic class, so I think that is why the initial Large class import is slow even after the Basic class has already been loaded.
Compiled assembly C# classes are the fastest to import every time, since the code has been pre-compiled.
In this post I looked at the usability implications of using PowerShell classes vs Inline C# classes. You can read the post for more details, but the takeaways are:
using module YourModule
instead of Import-Module YourModule
to be able to reference to the class and enum types, otherwise PowerShell will give an error that it cannot find the type.C# classes and enums, whether inline or compiled, do not have these limitations.
However, using C# code in PowerShell does have its own drawbacks:
See the list of C# versions and their features to know which C# features you can use.
If you are not creating a module, but instead just writing a script, I recommend using PowerShell classes. It avoids context switching between languages and is still very fast to run.
If you are creating a module and don’t intend for consumers of the module to use the classes and enums; that is, they will only be referenced internally by your module, then using PowerShell classes may still be fine.
If you are creating a module and intend to expose your classes and enums for consumers to use, then I recommend using C# classes and enums.
If the module load time is not a concern, then inline C# classes may be fine. If you add many inline C# classes though, it could take several seconds to load the module.
If module load time is a concern, then I recommend using compiled assembly C# classes and enums. Using a compiled assembly gives you the benefit of compiler checking and being able to use other development tools if you like. However, it also adds additional development complexity as you need to compile the assembly after any code changes and before you can use it in PowerShell. This is the approach I took in my tiPS PowerShell module, as it is intended to be added to a user’s PowerShell profile and would be loaded every session, so you can look at that module for an example.
Since you are not able to debug C# classes when running PowerShell, nor write to all of the output streams, I recommend keeping your classes very simple and using them mostly as a data structure. They should mostly just be properties with no or very few methods. If complex operations need to be performed on the class data, create PowerShell functions that accept the class as a parameter and perform the operation. This will allow you to step through and debug the complex code, and write to all of the output streams if needed.
It is possible to create modules and cmdlets entirely in C# instead of PowerShell. This gives you all of the benefits that come with writing C#: strongly typed code, compiler checking, great development tools, etc. For more information on how to do this, check out my other blog post.
In this article we’ve seen the pros, cons, and performance of the 3 different ways to define and import classes and enums in PowerShell. I hope this helps you decide which approach is best for your scenario.
If you have any questions or comments, please leave them below.
Happy coding!
]]>Install and configure the tiPS module by running the following commands in a PowerShell terminal:
Install-Module -Name tiPS -Scope CurrentUser
Add-TiPSImportToPowerShellProfile
Set-TiPSConfiguration -AutomaticallyWritePowerShellTip Daily -AutomaticallyUpdateModule Weekly
To display a PowerShell tip, simply run the Write-PowerShellTip
command, or use its alias tips
.
Here’s a quick demo of installing tiPS and getting tips on demand, and then configuring tiPS to automatically display a tip every time you open a new PowerShell session:
While the demo shows displaying a new tip on every PowerShell session, I recommend configuring it to show a new tip daily so that you don’t get too distracted by tips while doing your day-to-day work.
tiPS is cross-platform, so it works on Windows, macOS, and Linux. It supports both Windows PowerShell (e.g. v5.1) and PowerShell Core (e.g. v7+).
tiPS is open source and intended to be community driven. If you have a PowerShell tip, module, blog post, or community event that you think others would find useful, submit a pull request to have it added.
Checkout the tiPS GitHub repo for more information.
Happy scripting!
]]>Jakub Jareš has created the amazing Profiler module GitHub repo for profiling PowerShell code to find which parts are the slowest. Installing and using it is a breeze.
In this post I will walk through how I used Profiler to find a surprisingly slow part of a new PowerShell module I am developing, called tiPS. I noticed that importing tiPS in my PowerShell profile was noticeably slowing down my PowerShell session startup time, so I decided to profile it to see if I could optimize it.
If you don’t want to read this entire post, just run these commands and thank me (and Jakub) later:
Install-Module -Name Profiler
$trace = Trace-Script -ScriptBlock { & 'C:\Path\To\Script.ps1' }
$trace.Top50SelfDuration | Out-GridView
The Profiler module is available on the PowerShell Gallery here, so you can install it with:
Install-Module -Name Profiler
You profile code by tracing it, which is done with the Trace-Script
command.
This works similarly to the Measure-Command
cmdlet, but provides more detailed information.
To use the Profiler, simply wrap the code you want to profile in a script block and pass it to the Trace-Script
command, and capture the output in a variable.
Trace some code inline:
$trace = Trace-Script -ScriptBlock {
# Code to profile goes here
}
Trace some code in a script file:
$trace = Trace-Script -ScriptBlock { & 'C:\Path\To\Script.ps1' }
Trace some code in a script file that takes parameters:
$trace = Trace-Script -ScriptBlock { & 'C:\Path\To\Script.ps1' -Parameter 'Value' }
Trace the code in your $Profile
script that is run on every new session:
pwsh -NoProfile -NoExit { $trace = Trace-Script { . $Profile } }
Note: Depending on how you have your profile configured (MS docs), you may need to reference one of the other profile file paths, such as $Profile.CurrentUserAllHosts
.
Let’s walk through tracing the code I have in my profile script to see if I can reduce my PowerShell session startup time.
You will notice some summary information is output to the console when you run the Trace-Script
command, including the total Duration
, which is the same info that the Measure-Command
cmdlet would have given us.
Now that we have the trace information in the $trace
variable, you can inspect its properties to view the trace info.
The TotalDuration
property is the same TimeSpan that Measure-Command
would have given us:
The real value of the Profiler is in the Top50SelfDuration
and Top50Duration
properties, which is a list of the top 50 slowest commands in the trace.
The 2 properties show similar information, just sorted based on SelfDuration
or Duration
respectively.
SelfDuration
is the time spent in the command itself, not including any time spent in functions that it called.
This shows you the lines of code that are actually slow in the script.Duration
is the time spent in the command, including all other functions it called.
It shows the bigger pieces that are slow because they contain smaller slow pieces.A good strategy is to look at the Top50SelfDuration
first to see which specific lines of code are the slowest.
You can then look at the Top50Duration
to see which larger parts of your script take the most time, and see if it matches your expectations.
Here is the output of the Top50SelfDuration
property:
In this screenshot we can see that the top offender (the first row) is a line in the tiPS
module’s Configuration.ps1
file on line 8
, which is called once and taking 693 milliseconds to execute.
tiPS
is a module that I am currently developing and have added to my profile to automatically import it on every new session.
By default the Top50SelfDuration
and Top50Duration
display their output in a table view.
Unfortunately the number of columns displayed is based on your console window width.
If we make the console window wider, we can see 2 more very useful columns: Function
and Text
, which is the actual line of code that was executed.
Even with the wider console window, much of the text is still cut off.
To see the full text we have a couple options.
We can pipe the output to Format-List
:
$trace.Top50SelfDuration | Format-List
This is good for seeing details about each individual entry, but not great for viewing the entire list.
My preferred approach is to pipe the results to Out-GridView
:
$trace.Top50SelfDuration | Out-GridView
This allows me to see all of the columns, reorder and hide columns I don’t care about, as well as sort and filter the results.
Now that we have the Text
column, we can see that the top offender is a call to Add-Type -Language CSharp
.
Since tiPS
is my module, I know that line is being used to compile and import some inline C# code.
The 3rd row shows the same operation for importing a different C# file.
Since it is a single line of code calling a library that I do not own, there’s not much I can do to optimize it outside of considering a different strategy for importing C# code or not using C# at all, which may be something to consider.
Moving on, the 2nd row shows a call to Set-PoshPrompt
taking 624 milliseconds.
I use this third party module to customize my prompt with oh-my-posh.
In fact, looking at the Module
column, you can see many of the slowest commands are coming from 3 third party modules that I import in my profile script: oh-my-posh
, posh-git
, and Terminal-Icons
.
Since it is third party code there’s not much I can do outside of not loading those modules automatically at startup, which I may decide to do if they impact my startup time enough.
With that in mind, let’s focus on the code that I can optimize.
The 6th row shows another top offending line in the tiPS
module.
Notice that this line has a total Duration
of 1054 milliseconds, due to it being called 15 times.
On initial inspection, it appears that this line of code is slowing the module load time even more than our initial top offender (we’ll come back to this).
This is made more apparent when we sort by Duration
:
We can ignore the top row as it is dot-sourcing the file that contains all of my custom profile code, which is why the Percent
shows as 99.79
.
I keep my custom profile code in a separate file and dot-source it into the official $Profile so that it can be shared between PowerShell 5 and Pwsh 7, as well as automatically backed up on OneDrive.
Looking in my tiPS.psm1
file on line 25
, I see that it is looping over all of the files in the module directory and dot-sourcing them:
$functionFilePathsToImport | ForEach-Object {
. $_
}
This dot-sourcing strategy allows for better code organization by keeping module functions in separate files, rather than having one giant .psm1 file with all of the module code. A quick Google search shows me that others have come across this dot-sourcing performance issue as well. Naively we may think that this is giving additional evidence that the dot-sourcing may be the main issue.
Earlier we saw that the Add-Type
command was taking 693ms in Configuration.ps1, and 210ms in PowerShellTip.ps1.
That is 903ms total.
I did a Trace-Script -ScriptBlock { Import-Module -Name tiPS }
to see how long it takes to just import the tiPS module, and found the entire import takes 1180ms.
Configuration.ps1 and PowerShellTip.ps1 are 2 of the 15 files that get dot-sourced in the module, and they alone take 903ms of the 1180ms to load the module.
This highlights the difference between SelfDuration
and Duration
.
The Duration
of the dot-sourcing was including the time spent in all of the operations of the dot-sourced files, such as the Add-Type
operations.
In the screenshot we can see that the dot-sourcing Duration
is 1054ms, and the SelfDuration
is 129ms.
So the act of dot-sourcing the 15 files, regardless of what the files contain, is taking 129ms.
We know it takes 903ms to execute the two Add-Type
commands in the dot-sourced files, so that means the time it takes to execute the operations in the other 13 files being dot-sourced is 1054ms - 903ms = 151ms.
So dot-sourcing 15 files takes 129ms, which isn’t too bad.
I imagine (but did not test) the time will increase linearly with the number of files dot-sourced, so if you have a large project that dot-sources hundreds of files, you could be looking at several seconds of time spent just dot-sourcing the files; not executing the code in the files.
While dot-sourcing is a bit slow, it is not the main offender in my scenario.
The main performance issue is the Add-Type
commands to compile and import the C# code.
I never anticipated that one command would have such a drastic impact on the module load time!
Since the tiPS module is intended to be loaded on every new PowerShell session, I need to find a way to speed up the module load time. This might mean not using C# code, or taking some other approach; I’m not sure yet. The main point is that I now know where the bottleneck is and can focus my efforts on optimizing that code.
By using the Profiler module for less than 2 minutes I was able to identify a major performance issue that I would not have otherwise known about. Without it, I likely would have spent time optimizing other parts of my code that would not have as much impact.
Profiling your code is as easy as:
Install-Module -Name Profiler
then:
$trace = Trace-Script -ScriptBlock { & 'C:\Path\To\Script.ps1' }
$trace.Top50SelfDuration | Out-GridView
Profiler has more features that I did not cover in this post, such as using Flags to easily change the functionality of the code in the script block and compare them (e.g. to compare two different implementations), and functions for running the script block multiple times and taking the average performance metrics. Be sure to check out the Profiler git repo for more info.
I hope you’ve found this post helpful.
Happy profiling!
]]>