chill and drones 2

TAMT presents F-lithium - ...This Place Could Be My Undoing coverBad Sekta is a bit of an oddball British label releasing all sorts of electronic music, and recently they released 35 minutes of dark drone-y ambient you can download for free.
Since I like free music, I downloaded it through the online shop and gave it a whirl. Turns out this long-piece of dark ambient seems to be putting you into the metal bowls of spaceship flying through dark uncharted sectors of the galaxy. Or something like that. Anyway, it’s great to put up in the background and have it loop a few times, which you won’t notice, and get some work done.
So download Tamt presents F-Lithium, “…This Place Could Be My Undoing” and get spacin’.

Now that where at it, how about revisiting a good source for tons of ambient live mixes recorded to the trusty mp3 format? Remember the Chillits ambient festival? Well they have most of their live sets online, so you can pick and choose between hours of chilled beats, ambient sounds and other down-tempo tunes mixed and selected for your pleasure by top notch ambient DJ’s. If there is such a thing.

You can pick from the 2017 mixes, but mixes from previous editions are available too.
From the 2017 mixes, these are my mostly beat-less favorites: Brian Behlendorf, Dave Espionage, Indy Nyles, Jeremy Meadows, Souls and Cities, Subnaught, and Verdun 1916.
Also check my other ambient posts for more chilled goodness.

windows 10 upgrade on a dell xps 17 / L702x

I clean installed Windows 10 on my Dell XPS 17 L702x a while ago already and this post is merely here to indicate: yes, it works and it’s not even a big deal. This also got rid of Dell software junk preinstalled on my machine which is awesome.

Dell doesn’t support the Windows 10 upgrade though, which is why it scares people and your mileage may vary, but in my case the only thing that didn’t work anymore after the upgrade is my SD card reader. The drivers for that device haven’t been updated since the Windows 7 version and I didn’t find anything more recent. It took me a few months to find out it was broken, so that shows how much I need that thing.

So to summarize:

You can upgrade your OS or do a clean install with Windows 10, but your SD card reader will not work afterwards. If your life depends on that card slot on the left, it’s wise not to upgrade.

As always, be careful with drastic operations like this and backup your data first. For real this time. Plenty of stuff can go wrong and knowing that your data is safe makes it a lot less stressful if shit does happen to hit the fan.

querying Elasticsearch with Powershell with a little help from Kibana

KeyboardKibana is a great dash-boarding tool to easily query an Elasticsearch store on the fly without having to know exactly how to write an Elasticsearch query. For example if you’re using Logstash to dump all your logfiles into an Elasticsearch DB and use Kibana to nail down that specific weird exception you’re seeing.
Kibana is great to show some graphs and give a pretty good overview, but what if you want that query data and do some processing on that? You can’t really export it from the dashboard, but for each of those table or graph panels on your dashboard you can click the “Inspect” button and see what Elasticsearch query is used to get the data for the panel.

It looks something like this:

curl -XGET 'http://yourserver:9999/logstash_index_production/_search?pretty' -d '{
"query": { ...
}'

This is a curl statement and contains all you need to run the same query using PowerShell. The easiest thing to do is to copy the whole JSON statement into a text file and strip out the curl bit and the URL. You keep the URL handy because that’s the URL you’ll need to target in the Invoke-Restmethod call.
If you refactor it into something like the statements below and save it as a .ps1 file you can run it from the command-line and get the results back as PowerShell objects parsed from the JSON result. Yes. PowerShell is that cool. ;)

$elasticQuery = @"
{
"query": { ... }
}
"@

$elasticUri = 'http://yourserver:9999/logstash_index_production/_search?pretty'
Invoke-Restmethod -uri $elasticUri -method POST -Body $elasticQuery

To store the results in a local variable you just run it like this:

$r = .\RunElasticQuery.ps1

Now you’re free to run all sorts of funky processing on the data or perhaps dump it to a CSV file.

If you’re good enough at the Elasticsearch DSL you can even skip the Kibana query shortcut and modify the query itself in your PowerShell script.

Photo by Jeroen Bennink, cc-licensed.

stop the EU from destroying your internet freedom

On 12 September 2018 the EU Parliament will debate and vote on Article 13. Time to fire up your email and social media devices and let yourself be heard again, which is why I’m resurfacing this post about the what and the why.

Yep. They are at it again, those pesky governments. If it ain’t the US trying to destroy net neutrality it’s the EU trying to setup a link tax and an automated content filter/surveillance/censorship machine.

I’m talking about the copyright reform law the EU is trying to get through in a few days.

Article 11 is bad. It tries to setup a link tax, meaning you cant link or post snippets to e.g. news articles on your site. A similar law was passed earlier in Spain and it causes Google news to simply pull back out of Spain. If the same happens to the whole of the EU, that would suck mayor balls.

Article 13 is far worse though. That’s the content filter, which means any site where content can be uploaded e.g. Facebook, Twitter, Instagram, Imgur etc will be forced to automatically scan your upload and filter it if it isn’t allowed. The claim is that they want to stop terrorists and bad people from spreading illegal content on the internet. The reality will be that those bad guys will find ways around it and the rest of us will be stuck with a filter that’s going to block our uploads because of flawed algorithms and bureaucratic decisions. Internet memes use copyrighted content, but will the filter be able to detect sarcasm? I don’t think so.

Hey look, a meme, with a copyrighted image. I guess we won't be able to do that anymore once Article 13 is in effect.

To quote Tim Berners Lee, the inventor of the WWW:

Article 13 takes an unprecedented step towards the transformation of the Internet, from an open platform for sharing and innovation, into a tool for the automated surveillance and control of its users.

So please help out and email, tweet or call your MEP’s and make it clear Article 11 & 13 have to go. The freedom of the internet depends on you!

Update (25/6): the vote of the 20st was lost as 15 MEPS voted for article 13. Next date is 4-5 July to stop this thing from becoming final!
So let your voice be heard and save your internet!

Update (15/7): Yay!¬† We won! For now. After the summer break this will surely come back though. They won’t give up this easily.

invoke-webrequest pro tips

The Invoke-WebRequest PowerShell commandlet is great if you want to get and work with some web page’s output without installing any extra tools like wget.exe for example. If you’re planning to do some text parsing on a web page anyway, PowerShell is an excellent option, so why not go full PS mode?
Unfortunately the command has some drawbacks causing it to be a lot slower than it should be if you just want plain text and it’s response parsing can even cause it to lock up and not return a result at all.

So here’s some pro-tips for parsing the output using PowerShell fast and effectively:

1. Use basic parsing

The commandlet does some DOM parsing by default using Internet Explorer. This takes time and sometimes fails too, so if you want to skip this bit and make things faster, simply add the command-line switch UseBasicParsing:

$r = Invoke-WebRequest https://n3wjack.net -UseBasicParsing

2. Split html in lines

Parsing text in PS is easy, but it’s even easier if the result is formatted like a text file with multiple lines instead of the full HTML in a single string. If you get the Content property from your webpage, you can split it up into separate lines by splitting on the newline character:

(Invoke-WebRequest https://n3wjack.net -UseBasicParsing).Content -split "`n"

Or, if you also want the HTTP header info to be included in the result, use RawContent instead:

(Invoke-WebRequest https://n3wjack.net -UseBasicParsing).RawContent -split "`n"

This can be really handy if you want to automatically check if the right response headers are set.
But you can also use the Headers collection on the result object, which is even easier.

3. Disable download progress meter shizzle to download large files (or always to speed things up)

That download progress bar is a nice visual and all when you’re using Invoke-WebRequest to download some large binaries and want to see it’s progress, but it significantly slows things down too. Set the $progressPreference variable and you’ll see your scripts download those files a lot faster.
The larger the files (like big as log files, images, video’s etc) the more this matters I’ve noticed.

$progressPreference = 'silentlyContinue'
invoke-webrequest $logurl -outfile .\logfile.log -UseBasicParsing
$progressPreference = 'Continue'

Be sure to reactivate this setting afterwards, because this affects any commandlet using that progress-bar feature.

4. No redirects please.

Invoke-WebRequest automatically follows an HTTP redirect (301/302) so you end up with the page you where looking for in the most cases.
If you want to test if a URL is properly redirected (or not redirected) this just makes things harder. In that case you can turn off redirects by using the MaximumRedirection parameter and setting it to 0

When you get a URL that returns a 301 when doing this, the command will throw an exception saying the maximum redirection counts has been exceeded. This makes this case easier to test.
The result object will also contain the redirect StatusCode.

$r = Invoke-WebRequest http://n3wjack.net -MaximumRedirection 0

5. Use the PowerShell result object

It’s overkill in some cases, but in others this is pure win. The result object contains some really handy bits of the webpage, making a lot of tricky text and regex parsing obsolete.
It’s a piece of cake to parse all images linked from a page using the Image collection. Want to parse all outgoing links on a page? Use the Links collection. There’s also a StatusCode, a Headers collection a Forms and Inputfield collection for form parsing and more.
Check out what’s available using Get-Members:

Invoke-WebRequest https://n3wjack.net | get-members

4. If all else fails, use wget.exe

Yep. Sometimes Invoke-WebRequest simply doesn’t cut it. I’ve seen it hang on some complex pages trying to parse it and failing miserably.
In that case you get fetch the page using the GNU WGet tool, download the page as a text file and then parse that.
You have to call wget by adding the exe extension part otherwise you’ll be triggering the PowerShell alias for Invoke-WebRequest again.

# Install WGet with Chocolatey
choco install wget

# Get the page and save it as a text file
wget.exe https://n3wjack.net -O nj.html
# Read the file and parse it.
get-content nj.html | % { # parsing code goes here }

passing a function as a parameter in PowerShell

reflectionsA powerful pattern in software engineering is where you pass in a function or an object to another, which executes this given one dynamically. This allows you to extend behavior of code without having to edit that code. It’s known as the Strategy Pattern and allows for nice, clean and decoupled code.

In Powershell you can do this by passing a function as an argument to another function. When I tried to do this, I found out this wasn’t as trivial as I thought it would be, so here’s the nitty gritty on to work that magical extend-ability pattern.

Basically you want to do something like passing in a function which processes a single item in a loop controlled by another piece of code. Something like:

function Print-Number($number)
{
    echo "Number is $number"
}

function Do-Loop($function)
{
    $numbers = 1..10
    foreach ($number in $numbers)
    {
        # Here we should call $function and pass in $number
    }    
}

# Call Do-Loop with the PrintNumber function as a parameter here.

I ran into 2 problems here.

  1. How do I pass in a function as a reference to another one?
  2. How do I call one of those function blocks from the function that receives it?

First things first. The syntax for passing in a function inline looks like this:

Do-Loop ${function:Print-Number}

Note the function: prefix there. It’s the magic bit. This accesses the function object’s script block without executing it.
You can also list all functions available in your PowerShell session like this:

ls function:

If you want to pass in a script block like an anonymous lamba C# style, without defining a function first, you can do this:

Do-Loop { param($content) write-host $content }

Want to reuse that function a few times and store it in a variable? No problemo, just do this:

$function = { param($content) write-host $content }
Do-Loop $function
Do-SomethingElse $function

That pretty much covers all the options for problem number 1.
So now on the second problem: calling that passed in function or script block inside our host function.

Let’s say we want to call that function from a loop. To call the function you need to use the Invoke-Command commandlet and pass in the argument using the ArgumentList parameter like this:

foreach ($number in $numbers)
{
    Invoke-Command $function -ArgumentList $number
}

Pretty simple right?
The argument list expects an array as it’s value. So if you want to pass in 2 parameters like a number and a message text that would look something like this:

Invoke-Command $function -ArgumentList $number, $message

Putting it all together, here’s the working sample code:

function Print-Number($number)
{
    echo "Number is $number"
}

function Do-Loop($function)
{
    $numbers = 1..3
    foreach ($number in $numbers)
    {
        Invoke-Command $function -ArgumentList $number
    }    
}

Do-Loop ${function:\Print-Number}

Because of this array-as-a-parameter thing however I did run into a little snag for my actual code.
What if the first and only parameter is an array in itself? How do make it clear to the Invoke-Command commandlet that the array is a single parameter, not a list of parameters to pass into the function?

In my case I was passing in the content of a text file which is an array of strings. My first argument ended up being the first string of that array and I was lacking the rest of the file’s lines.

$array = Get-Content .\somefile.txt
Invoke-Command $function -ArgumentList $array # ?????

Again, there’s a little trick to that which I found on Stack Overflow. The syntax to pass in an array as a parameter is:

Invoke-Command $function -ArgumentList (,data)

The last bit seems to work by creating an array (that comma) where the first element is nothing and the second is our array.
Apparently the first element is then skipped and our second is passed in as the required array parameter.
A silly example to demonstrate this:

function Enhance($lines)
{
    $lines | % { "  > $_" }
}

function Do-It($function)
{
    $content = get-content .\awesome.txt
    Invoke-Command $function -ArgumentList (,$content)
}

Do-It ${function:\Enhance}