Category Archives: geek

invoke-webrequest pro tips

The Invoke-WebRequest PowerShell commandlet is great if you want to get and work with some web page’s output without installing any extra tools like wget.exe for example. If you’re planning to do some text parsing on a web page anyway, PowerShell is an excellent option, so why not go full PS mode?
Unfortunately the command has some drawbacks causing it to be a lot slower than it should be if you just want plain text and it’s response parsing can even cause it to lock up and not return a result at all.

So here’s some pro-tips for parsing the output using PowerShell fast and effectively:

1. Use basic parsing

The commandlet does some DOM parsing by default using Internet Explorer. This takes time and sometimes fails too, so if you want to skip this bit and make things faster, simply add the command-line switch UseBasicParsing:

$r = Invoke-WebRequest https://n3wjack.net -UseBasicParsing

2. Split html in lines

Parsing text in PS is easy, but it’s even easier if the result is formatted like a text file with multiple lines instead of the full HTML in a single string. If you get the Content property from your webpage, you can split it up into separate lines by splitting on the newline character:

(Invoke-WebRequest https://n3wjack.net -UseBasicParsing).Content -split "`n"

Or, if you also want the HTTP header info to be included in the result, use RawContent instead:

(Invoke-WebRequest https://n3wjack.net -UseBasicParsing).RawContent -split "`n"

This can be really handy if you want to automatically check if the right response headers are set.
But you can also use the Headers collection on the result object, which is even easier.

3. Disable download progress meter shizzle to download large files (or always to speed things up)

That download progress bar is a nice visual and all when you’re using Invoke-WebRequest to download some large binaries and want to see it’s progress, but it significantly slows things down too. Set the $progressPreference variable and you’ll see your scripts download those files a lot faster.
The larger the files (like big as log files, images, video’s etc) the more this matters I’ve noticed.

$progressPreference = 'silentlyContinue'
invoke-webrequest $logurl -outfile .\logfile.log -UseBasicParsing
$progressPreference = 'Continue'

Be sure to reactivate this setting afterwards, because this affects any commandlet using that progress-bar feature.

4. No redirects please.

Invoke-WebRequest automatically follows an HTTP redirect (301/302) so you end up with the page you where looking for in the most cases.
If you want to test if a URL is properly redirected (or not redirected) this just makes things harder. In that case you can turn off redirects by using the MaximumRedirection parameter and setting it to 0

When you get a URL that returns a 301 when doing this, the command will throw an exception saying the maximum redirection counts has been exceeded. This makes this case easier to test.
The result object will also contain the redirect StatusCode.

$r = Invoke-WebRequest http://n3wjack.net -MaximumRedirection 0

5. Use the PowerShell result object

It’s overkill in some cases, but in others this is pure win. The result object contains some really handy bits of the webpage, making a lot of tricky text and regex parsing obsolete.
It’s a piece of cake to parse all images linked from a page using the Image collection. Want to parse all outgoing links on a page? Use the Links collection. There’s also a StatusCode, a Headers collection a Forms and Inputfield collection for form parsing and more.
Check out what’s available using Get-Members:

Invoke-WebRequest https://n3wjack.net | get-members

4. If all else fails, use wget.exe

Yep. Sometimes Invoke-WebRequest simply doesn’t cut it. I’ve seen it hang on some complex pages trying to parse it and failing miserably.
In that case you get fetch the page using the GNU WGet tool, download the page as a text file and then parse that.
You have to call wget by adding the exe extension part otherwise you’ll be triggering the PowerShell alias for Invoke-WebRequest again.

# Install WGet with Chocolatey
choco install wget

# Get the page and save it as a text file
wget.exe https://n3wjack.net -O nj.html
# Read the file and parse it.
get-content nj.html | % { # parsing code goes here }

passing a function as a parameter in PowerShell

reflectionsA powerful pattern in software engineering is where you pass in a function or an object to another, which executes this given one dynamically. This allows you to extend behavior of code without having to edit that code. It’s known as the Strategy Pattern and allows for nice, clean and decoupled code.

In Powershell you can do this by passing a function as an argument to another function. When I tried to do this, I found out this wasn’t as trivial as I thought it would be, so here’s the nitty gritty on to work that magical extend-ability pattern.

Basically you want to do something like passing in a function which processes a single item in a loop controlled by another piece of code. Something like:

function Print-Number($number)
{
    echo "Number is $number"
}

function Do-Loop($function)
{
    $numbers = 1..10
    foreach ($number in $numbers)
    {
        # Here we should call $function and pass in $number
    }    
}

# Call Do-Loop with the PrintNumber function as a parameter here.

I ran into 2 problems here.

  1. How do I pass in a function as a reference to another one?
  2. How do I call one of those function blocks from the function that receives it?

First things first. The syntax for passing in a function inline looks like this:

Do-Loop ${function:Print-Number}

Note the function: prefix there. It’s the magic bit. This accesses the function object’s script block without executing it.
You can also list all functions available in your PowerShell session like this:

ls function:

If you want to pass in a script block like an anonymous lamba C# style, without defining a function first, you can do this:

Do-Loop { param($content) write-host $content }

Want to reuse that function a few times and store it in a variable? No problemo, just do this:

$function = { param($content) write-host $content }
Do-Loop $function
Do-SomethingElse $function

That pretty much covers all the options for problem number 1.
So now on the second problem: calling that passed in function or script block inside our host function.

Let’s say we want to call that function from a loop. To call the function you need to use the Invoke-Command commandlet and pass in the argument using the ArgumentList parameter like this:

foreach ($number in $numbers)
{
    Invoke-Command $function -ArgumentList $number
}

Pretty simple right?
The argument list expects an array as it’s value. So if you want to pass in 2 parameters like a number and a message text that would look something like this:

Invoke-Command $function -ArgumentList $number, $message

Putting it all together, here’s the working sample code:

function Print-Number($number)
{
    echo "Number is $number"
}

function Do-Loop($function)
{
    $numbers = 1..3
    foreach ($number in $numbers)
    {
        Invoke-Command $function -ArgumentList $number
    }    
}

Do-Loop ${function:\Print-Number}

Because of this array-as-a-parameter thing however I did run into a little snag for my actual code.
What if the first and only parameter is an array in itself? How do make it clear to the Invoke-Command commandlet that the array is a single parameter, not a list of parameters to pass into the function?

In my case I was passing in the content of a text file which is an array of strings. My first argument ended up being the first string of that array and I was lacking the rest of the file’s lines.

$array = Get-Content .\somefile.txt
Invoke-Command $function -ArgumentList $array # ?????

Again, there’s a little trick to that which I found on Stack Overflow. The syntax to pass in an array as a parameter is:

Invoke-Command $function -ArgumentList (,data)

The last bit seems to work by creating an array (that comma) where the first element is nothing and the second is our array.
Apparently the first element is then skipped and our second is passed in as the required array parameter.
A silly example to demonstrate this:

function Enhance($lines)
{
    $lines | % { "  > $_" }
}

function Do-It($function)
{
    $content = get-content .\awesome.txt
    Invoke-Command $function -ArgumentList (,$content)
}

Do-It ${function:\Enhance}

getting rid of the McAfee Process Validation Service which I never installed

Recently I noticed my laptop was acting rather sluggish after rebooting. As a Windows geek I swiftly started the improved Windows 10 task manager and noticed something peculiar.

A “McAfee Process Validation Service” (or mfevtps.exe for you techies) was gobbling up a lot of CPU cycles. I know McAfee of course, but I never installed any of their antivirus products, so how on earth did that get on my system?

A screenshots showing the McAfee service scanning and slowing down my system why I didn't even install it. How queer.

I found out that this thing is actually a Windows service, which you can see in the Service tab of your Task Manager. So when you look into the startup tab to see what triggers the process, you don’t even see it there.

It also turns out that this bit of software is automatically installed when you run the McAfee Stinger antivirus detection software.
Now Stinger is great, or at least used to be great. It’s supposed to be a standalone executable you can just pop onto any system and use it to scan for viruses and malware without having to install any full blow antivirus suite.

Unfortunately this now also installs a bit of malware itself.

Now how do I get rid of something that I never installed? It can’t be found anywhere when you use the regular Windows uninstall tools. I knew this thing was sitting in my c:\windows\system32 folder, but I didn’t want to just rip it out by hand because there might still be some other crap littered here and there that I don’t know about, and might be causing problems once it’s half destroyed.

If found out that McAfee has a specific removal tool. I guess they get this question a lot…
Finally this MCPR.exe removal tool deleted the unwanted service and after a single reboot my system now is a tiny bit cleaner and a tiny bit faster again.

Which is nice.

how to stop windows 10 from rearranging your desktop icons

I’m a desktop minimalist myself and the trashcan is about all I want to see on my desktop. However if you like your desktop with lots of icons on it, arranged according to some non-alphabetical grouping system you might find out that Windows sometimes rearranges them nicely in alphabetical order, aligning them all nicely to the top left of your desktop without your prior consent.
This typically happens after reboots, switching between resolutions, plugging and unplugging a secondary screen or docking/un-docking the laptop.
It can drive people mad.

This seems to be an issue that has been plaguing Windows 8 and 10 for quite a while already, driving lot’s of people bonkers. Unfortunately when looking for a solution online in various fora you tend to find a lot of useless tips or the suggestion to install some third-party desktop icon position-saving software. Eew.

There is a simple solution in the Windows settings however, but I have to admit it’s not very obvious and I had to look for it quite a bit to find it.

It’s all in the settings as expected, but it’s spread out in two different spots, which makes it hard to get right.

Here’s how to fix it:

1. Right click on your desktop and disable the auto arrange feature by disabling the checkbox next in View > Auto arrange icons.

Shows how to disable the auto arrange icons feature in Windows.

You’d think that should do the trick right? Well most people do. But there’s another option that interferes with your desktop icon zen garden staying the way you want it.

2. Right click again on your desktop, now click Personalize.
You’ll see the Settings screen appear.
There you select Themes.
Now click Desktop icon settings.
Now look what we have there. A checkbox labeled “Allow themes to change desktop icons“.

Uncheck it and press OK.

You should now no longer experience the frustration of computer code messing with your desktop icons.

Zen at last.

Disable the Allow themes to change desktop icons checkbox

 

fixing MSB3644 build errors and the point of .NET targeting packs

Build errors on build servers suck. If it builds locally, why the hell doesn’t it build on the build server? Well there’s plenty of reasons for that, but as a .NET developer is usually means that something you have on your machine that came with your Visual Studio install isn’t installed on the build server.
But you probably don’t want to install the full-blown VS on the build server, so the question now is: what bit do I need to install?

Recently I ran into the following build error on 1 specific build server (yep, not on another one, fun, fun, fun).

C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\bin\Microsoft.Common.CurrentVersion.targets(1111, 5): error MSB3644: The reference assemblies for framework ".NETFramework,Version=v4.5.2" were not found. To resolve this, install the SDK or Targeting Pack for this framework version or retarget your application to a version of the framework for which you have the SDK or Targeting Pack installed. Note that assemblies will be resolved from the Global Assembly Cache (GAC) and will be used in place of reference assemblies. Therefore your assembly may not be correctly targeted for the framework you intend.

It clearly has something to do with a project which targets .NET framework v4.5.2.
The answer in fact is right there. You need to install a “targeting pack”.
But WTF is that thing and why do I need it?

Apparently having the .NET framework itself installed on your machine isn’t enough to be able to build a project if that project targets a specific version of the .NET framework. It makes sense as the .NET framework installation is actually the runtime, used to run .NET applications, not build them.
If you want to build apps against a specific version of the .NET framework, you need that targeting pack as well on your build machine. This is either your development box, which has VS on it, and thus the required targeting packs which come with the VS installation. Or this can be your build server, where you have to install the targeting packs as well.

You can check what targeting packs you already have on a machine by checking the sub-folders in “C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework
For each supported framework version, there’s a v<version number> folder there. In my case there was no v4.5.2 folder on that one machine.

So where do you find those targeting packs for the various .NET framework versions? Microsoft luckily compiled a nice list where you can find all the download links and instructions.
For the versions listed as “included in VS 2017”, you can see them all listed in the VS 2017 installer if you go to the “Individual components” tab.

Look at all those .NET targeting packs.

So a shortcut to install all the packs at once, is to grab the VS2017 installer and use that one. But you might want to disable all the IDE specific stuff you won’t need on the build server though.

cure cancer with leftover azure credits

An azure window grid, reflecting clouds. Perfect fit isn't it?Maybe you have some Azure credits lying from a Visual Studio subscription you have from work, waiting to be spent on cool and nifty experiments, but don’t end up actually using them.
How about spending some of those dollars on cancer research? Or help find a cure against Zika? Fighting AIDS maybe?

Enter the World Community Grid, a vast grid computing network running on the Open Source BOINC client software, started as a philanthropic initiative by IBM.
Sounds good right?

All you need to do is create a WCG account, spin up a Linux machine on Azure, install the BOINC headless client on it and link it to your account. In about half an hour you’ll be computing cancer markers, folding genes or fighting some horrible disease. Well, the software will be doing that really, which is even better.

Here we go, step by step.

1. Setup your Azure Linux machine, I chose an Ubuntu 16 LTS machine. I pick the classic VM because its way easier to setup.
Depending on the type of machine you’ll have more compute power and thus turning out more results. Try a few out and see how you can maximize your Azure credits.

2. Once provisioned, log in using PUTTY or your favorite SSH client. Now it’s time to update the Linux packages and then install the BOINC client:

sudo apt-get update
sudo apt-get install boinc-client

3. Setup auto startup of the BOINC client, so if your machine reboots, you don’t have to go in and start it up yourself (automate all the things remember):

sudo /etc/init.d/boinc-client restart

4. Get your BOINC authentication key so you can hook up the client to your account:

boinccmd --lookup_account https://www.worldcommunitygrid.org

5. Now use the key to attach your selected projects from WCG:

boinccmd --project_attach https://www.worldcommunitygrid.org

Of course you want to check what’s going on, so you can check the BOINC client’s state like this:

boinccmd --get_state

This should give you a list of the tasks and their state. It might take a while before they start kicking in, but you’ll see results coming in after a day or so on the WGC website under your contribution history.

That’s about it. Your client is all set up. All you need to do is keep those VM’s running in the cloud, which normally takes non effort at all. Neat.

There are more boinccmd command line switches in the documentation if you’d need to troubleshoot or find out more.

What’s next? Well, you can set up more than 1 machine if you like, or a heavier one and see what gives you more bang for your buck.¬† You can also join my World Computing Grid team called “Team Azure” and see how many cloud bucks we can burn. It’ll be effortless fun, I promise! ;)

Credit goes to Joel Christian’s headless Ubuntu installation guide. His guide made my quest to setup BOINC on an Azure Ubuntu box a lot easier.

Photo by Dan, cc-licensed.