Ox makes words. Sometimes2023-02-04T06:50:52+00:00https://oxo42.github.ioJohn OxleyUsing future Streams in Rust with Results2023-02-04T00:00:00+00:00https://oxo42.github.io/2023/02/04/using-future-streams-in-rust-with-results
<h1 id="using-future-streams-in-rust-with-results">Using future <code class="language-plaintext highlighter-rouge">Stream</code>s in Rust with <code class="language-plaintext highlighter-rouge">Result</code>s</h1>
<p>Have you ever wanted to do something in parallel in Rust? In a recent project, I wanted to fetch a list of things from a network server then for each thing make another request to the server. Well some Googling brought up <a href="https://www.reddit.com/r/rust/comments/ueyt1d/comment/i6qsois/?utm_source=reddit&utm_medium=web2x&context=3">this reddit comment</a>. This is a story of the journey from that comment to the diff and the problems I hit along the way.</p>
<p>I’m going to talk about using the <a href="https://docs.rs/futures/latest/futures/index.html">futures</a> crate, specifically <a href="https://docs.rs/futures/latest/futures/stream/index.html">streams</a> which represent a series of values asynchronously. Things went well until <a href="https://doc.rust-lang.org/std/result/enum.Result.html"><code class="language-plaintext highlighter-rouge">Result</code></a> came into the picture</p>
<h2 id="syncronous-maps-and-result">Syncronous maps and Result</h2>
<p>Let’s say we have a fallible (I’m not <a href="https://doc.rust-lang.org/rust-by-example/error/result.html">going to explain Result here</a>) function</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">double_sync</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">usize</span><span class="p">)</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="nb">usize</span><span class="p">,</span> <span class="nb">String</span><span class="o">></span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="mi">3</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">Err</span><span class="p">(</span><span class="s">"I don't like three"</span><span class="nf">.to_owned</span><span class="p">())</span>
<span class="err"> </span> <span class="err"> </span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">Ok</span><span class="p">(</span><span class="n">x</span> <span class="o">*</span> <span class="mi">2</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>and you want to apply it to a range of <code class="language-plaintext highlighter-rouge">usize</code>’s, you can use iterators and <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.map"><code class="language-plaintext highlighter-rouge">map</code></a> then <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.collect"><code class="language-plaintext highlighter-rouge">collect</code></a> them into a <code class="language-plaintext highlighter-rouge">Vec<Result<T, E>></code>. Let’s try</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span><span class="n">double_sync</span><span class="p">)</span><span class="nf">.collect</span><span class="p">();</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<p>gives</p>
<pre><code class="language-plain">[Ok(0), Ok(2), Ok(4), Err("I don't like three"), Ok(8)]
</code></pre>
<p>A few things to explain:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">collect</code> needs a hint of what collection to build. You can do what I’ve done or use the turbofish syntax <code class="language-plaintext highlighter-rouge">.collect::<Vec<_></code>. The same thing happens in either case, it’s just where you want to put things
<ul>
<li>One of my favorite crates, <a href="https://docs.rs/itertools/latest/itertools/index.html">Itertools</a> has <a href="https://docs.rs/itertools/latest/itertools/trait.Itertools.html#method.collect_vec"><code class="language-plaintext highlighter-rouge">collect_vec</code></a> which is <code class="language-plaintext highlighter-rouge">collect</code> for Vectors (for the impatient, look at the next method)</li>
</ul>
</li>
<li>The <code class="language-plaintext highlighter-rouge">_</code> tells rust, “Hey whatever map returns, I want a collection of that thing” and in our case it’s a <code class="language-plaintext highlighter-rouge">Result<usize, String></code></li>
<li><code class="language-plaintext highlighter-rouge">0..5</code> is a <a href="https://doc.rust-lang.org/std/ops/struct.Range.html">Range</a> which does not include <code class="language-plaintext highlighter-rouge">5</code>.
<ul>
<li><code class="language-plaintext highlighter-rouge">0..=5</code> would be an <a href="https://doc.rust-lang.org/std/ops/struct.RangeInclusive.html">Inclusive Ranges</a></li>
<li>Ranges are Iterators</li>
</ul>
</li>
</ul>
<p>So this is good. I get all the results back, but I have a mix of <code class="language-plaintext highlighter-rouge">Ok</code>s and <code class="language-plaintext highlighter-rouge">Err</code>s. What if I wanted to turn things from a <code class="language-plaintext highlighter-rouge">Vec<Result<T, E>></code> into a <code class="language-plaintext highlighter-rouge">Result<Vec<T>, E></code>. So either <code class="language-plaintext highlighter-rouge">Ok([ 0, 2, 4])</code> or <code class="language-plaintext highlighter-rouge">Err("blah")</code>. Well because <code class="language-plaintext highlighter-rouge">Result</code> implements <a href="https://doc.rust-lang.org/std/result/enum.Result.html#method.iter"><code class="language-plaintext highlighter-rouge">iter</code></a>, collect can <em>magic</em> things around for us. And if you’re reading this I decided not to come back and explain said magic. Oh too bad.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="n">Result</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span><span class="p">,</span> <span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span><span class="n">double_sync</span><span class="p">)</span><span class="nf">.collect</span><span class="p">();</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<pre><code class="language-plain">Err("I don't like three")
</code></pre>
<p>hmm… so all the <code class="language-plaintext highlighter-rouge">Ok</code>s are discarded, but you can <a href="https://doc.rust-lang.org/rust-by-example/error/iter_result.html#collect-the-failed-items-with-map_err-and-filter_map">keep the errors if you want to</a>. Fine what about all successful.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="n">Result</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span><span class="p">,</span> <span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="p">(</span><span class="mi">4</span><span class="o">..=</span><span class="mi">6</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span><span class="n">double_sync</span><span class="p">)</span><span class="nf">.collect</span><span class="p">();</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Ok([8, 10, 12])
</code></pre></div></div>
<h2 id="what-about-async">What about async?</h2>
<p>I’m going to start of with a function that can’t fail</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">async</span> <span class="k">fn</span> <span class="nf">double</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">usize</span><span class="p">)</span> <span class="k">-></span> <span class="nb">usize</span> <span class="p">{</span>
<span class="n">x</span> <span class="o">*</span> <span class="mi">2</span>
<span class="p">}</span>
</code></pre></div></div>
<p>How do I map that across my input asynchronously. Well you can turn the iterator into a <a href="https://docs.rs/futures/latest/futures/stream/trait.Stream.html"><code class="language-plaintext highlighter-rouge">Stream</code></a> with <a href="https://docs.rs/futures/latest/futures/stream/fn.iter.html">‘futures::stream::iter’</a>.</p>
<p>Something that future-Ox has just told me will be important, a Stream is just like an iterator but instead of <code class="language-plaintext highlighter-rouge">next()</code> it has a <a href="https://docs.rs/futures/latest/futures/stream/trait.Stream.html"><code class="language-plaintext highlighter-rouge">poll_next</code></a> function that returns a <a href="https://docs.rs/futures/latest/futures/task/enum.Poll.html"><code class="language-plaintext highlighter-rouge">futures::task::Poll</code></a> enum that the async runtime can come back to. Read <a href="https://fasterthanli.me/articles/understanding-rust-futures-by-going-way-too-deep">Understanding Rust futures by going way too deep</a> for why this matters.</p>
<p>Let’s do this</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">usize</span><span class="o">></span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span><span class="n">double_infallible</span><span class="p">)</span><span class="nf">.collect</span><span class="p">();</span>
</code></pre></div></div>
<p>Go! Rust! Go… err what</p>
<pre><code class="language-rust_errors">error[E0599]: `futures::stream::Iter<std::ops::Range<{integer}>>` is not an iterator
--> src/main.rs:33:10
|
33 | .map(double_infallible)
| ^^^ `futures::stream::Iter<std::ops::Range<{integer}>>` is not an iterator
|
::: /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/stream/iter.rs:9:1
|
9 | pub struct Iter<I> {
| ------------------ doesn't satisfy `_: Iterator`
|
= note: the following trait bounds were not satisfied:
`futures::stream::Iter<std::ops::Range<{integer}>>: Iterator`
which is required by `&mut futures::stream::Iter<std::ops::Range<{integer}>>: Iterator`
= help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope; perhaps add a `use` for it:
|
4 | use futures::StreamExt;
</code></pre>
<p><code class="language-plaintext highlighter-rouge">futures:stream::Iter</code> is not an iterator. Wat! Oh wait hold on, we need to bring in the <a href="http://xion.io/post/code/rust-extension-traits.html">Extension Trait</a> <a href="https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html"><code class="language-plaintext highlighter-rouge">StreamExt</code></a> which actually has the <a href="https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.map"><code class="language-plaintext highlighter-rouge">map</code></a> function. And <code class="language-plaintext highlighter-rouge">rustc</code> kindle tells us to <code class="language-plaintext highlighter-rouge">use futures::StreamExt</code>. Here we go again</p>
<pre><code class="language-rust_errors">error[E0308]: mismatched types
--> src/main.rs:32:31
|
32 | let results: Vec<usize> = futures::stream::iter(0..5)
| __________________----------___^
| | |
| | expected due to this
33 | | .map(double_infallible)
34 | | .collect();
| |__________________^ expected struct `Vec`, found struct `Collect`
|
= note: expected struct `Vec<usize>`
found struct `Collect<futures::stream::Map<futures::stream::Iter<std::ops::Range<usize>>, fn(usize) -> impl futures::Future<Output = usize> {double_infallible}>, _>`
</code></pre>
<p>What now? So, <a href="https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.collect"><code class="language-plaintext highlighter-rouge">collect</code></a> returns a <a href="https://docs.rs/futures/latest/futures/stream/struct.Collect.html"><code class="language-plaintext highlighter-rouge">Collect<St, C></code></a>. Let’s look at <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/collect.rs.html">the source for Collect</a>. We have</p>
<ul>
<li>The <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/collect.rs.html#13">struct definition</a> Line 13</li>
<li>The <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/collect.rs.html#13">implementation for the struct</a> where it ensures that
<ul>
<li><code class="language-plaintext highlighter-rouge">St</code> implements <code class="language-plaintext highlighter-rouge">Stream</code></li>
<li><code class="language-plaintext highlighter-rouge">C</code> implements <code class="language-plaintext highlighter-rouge">Default</code>.</li>
</ul>
</li>
<li>A <a href="https://docs.rs/futures/latest/futures/future/trait.FusedFuture.html">FusedFuture</a> <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/collect.rs.html#30">implementation</a> which says the <code class="language-plaintext highlighter-rouge">Collect</code> is terminated and can no longer be polled</li>
<li>And the good stuff, an <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/collect.rs.html#40">implementation of Future for Collect</a>. This means we can <code class="language-plaintext highlighter-rouge">.await</code> it. Lets try</li>
</ul>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">usize</span><span class="o">></span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">.map</span><span class="p">(</span><span class="n">double_infallible</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">.collect</span><span class="p">()</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="k">.await</span><span class="p">;</span>
</code></pre></div></div>
<p>Holy crap!</p>
<pre><code class="language-rust_errors">error[E0277]: the trait bound `Vec<usize>: Extend<impl futures::Future<Output = usize>>` is not satisfied
--> src/main.rs:35:9
|
35 | .await;
| ^^^^^^ the trait `Extend<impl futures::Future<Output = usize>>` is not implemented for `Vec<usize>`
|
= help: the following other types implement trait `Extend<A>`:
<Vec<T, A> as Extend<&'a T>>
<Vec<T, A> as Extend<T>>
= note: required for `Collect<futures::stream::Map<futures::stream::Iter<std::ops::Range<usize>>, ...>, ...>` to implement `futures::Future`
= note: the full type name has been written to '/playground/target/debug/deps/playground-2814c18f3846befb.long-type-4383195030087971247.txt'
error[E0277]: the trait bound `Vec<usize>: Extend<impl futures::Future<Output = usize>>` is not satisfied
--> src/main.rs:32:31
|
32 | let results: Vec<usize> = futures::stream::iter(0..5)
| _______________________________^
33 | | .map(double_infallible)
| |_______________________________^ the trait `Extend<impl futures::Future<Output = usize>>` is not implemented for `Vec<usize>`
34 | .collect()
| ------- required by a bound introduced by this call
|
= help: the following other types implement trait `Extend<A>`:
<Vec<T, A> as Extend<&'a T>>
<Vec<T, A> as Extend<T>>
note: required by a bound in `futures::StreamExt::collect`
--> /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/stream/stream/mod.rs:516:29
|
516 | fn collect<C: Default + Extend<Self::Item>>(self) -> Collect<Self, C>
| ^^^^^^^^^^^^^^^^^^ required by this bound in `futures::StreamExt::collect`
For more information about this error, try `rustc --explain E0277`.
</code></pre>
<p>(ノಠ益ಠ)ノ彡┻━┻</p>
<h2 id="pull-apart-the-stream">Pull apart the Stream</h2>
<p>Okay let’s go back. What does the map actually give us. Excuse me <code class="language-plaintext highlighter-rouge">rustc</code> how do you barf on this</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="nb">i32</span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span><span class="n">double_infallible</span><span class="p">);</span>
</code></pre></div></div>
<p>And without long module names</p>
<pre><code class="language-rust_errors">note: expected type `i32`
found struct `Map<Iter<Range<usize>>, fn(usize) -> impl Future<Output = usize> {double_infallible}>`
</code></pre>
<p>Aha, so in <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/map.rs.html#12-20"><code class="language-plaintext highlighter-rouge">Map</code></a> where the <code class="language-plaintext highlighter-rouge">St</code>ream is a <code class="language-plaintext highlighter-rouge">FusedStream</code> and the function takes a <code class="language-plaintext highlighter-rouge">usize</code> and returns a <code class="language-plaintext highlighter-rouge">Future</code>. Ahhh right, so when we collect this, we’ll actuallly get collection of <code class="language-plaintext highlighter-rouge">Future</code> not a collection of <code class="language-plaintext highlighter-rouge">usize</code>. So back to my collect, let’s try with some inference</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">.map</span><span class="p">(</span><span class="n">double_infallible</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">.collect</span><span class="p">()</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="k">.await</span><span class="p">;</span>
</code></pre></div></div>
<p>Yay! It compiles. Now from what we looked at with <code class="language-plaintext highlighter-rouge">Map</code> I betcha it’s a <code class="language-plaintext highlighter-rouge">Vec</code> of Future that produces a <code class="language-plaintext highlighter-rouge">usize</code>. Let’s <code class="language-plaintext highlighter-rouge">println!("{results:?}")</code></p>
<pre><code class="language-rust_errors">error[E0277]: `impl futures::Future<Output = usize>` doesn't implement `Debug`
--> src/main.rs:43:16
|
43 | println!("{results:?}");
| ^^^^^^^ `impl futures::Future<Output = usize>` cannot be formatted using `{:?}` because it doesn't implement `Debug`
|
= help: the trait `Debug` is not implemented for `impl futures::Future<Output = usize>`
= help: the trait `Debug` is implemented for `Vec<T, A>`
= note: this error originates in the macro `$crate::format_args_nl` which comes from the expansion of the macro `println` (in Nightly builds, run with -Z macro-backtrace for more info)
</code></pre>
<p>Where the error is happening is unclear (to me), but I think that’s right. What does <a href="https://stackoverflow.com/questions/21747136/how-do-i-print-in-rust-the-type-of-a-variable"><code class="language-plaintext highlighter-rouge">print_type_of</code></a> say?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>alloc::vec::Vec<playground::double_infallible::>
</code></pre></div></div>
<p>Heey, there we go. So how do I execute those futures? Let’s do it the ourselves</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">mut</span> <span class="n">values</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">usize</span><span class="o">></span> <span class="o">=</span> <span class="nn">Vec</span><span class="p">::</span><span class="nf">with_capacity</span><span class="p">(</span><span class="n">results</span><span class="nf">.len</span><span class="p">());</span>
<span class="k">for</span> <span class="n">f</span> <span class="n">in</span> <span class="n">results</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="n">values</span><span class="nf">.push</span><span class="p">(</span><span class="n">f</span><span class="k">.await</span><span class="p">);</span>
<span class="p">}</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{values:?}"</span><span class="p">);</span>
</code></pre></div></div>
<p>Whoop!!!!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0, 2, 4, 6, 8]
</code></pre></div></div>
<p>┳━┳ ヽ(ಠل͜ಠ)ノ
But surely, something can do this for us. Why yes, indeed</p>
<h2 id="buffered-and-buffer_unordered"><a href="https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.buffered">buffered</a> and <a href="https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.buffer_unordered">buffer_unordered</a></h2>
<p>Both of these have a very important</p>
<blockquote>
<p>The returned stream will be a stream of each future’s output</p>
</blockquote>
<p>in the docs. Remember when I said a <code class="language-plaintext highlighter-rouge">Stream</code> has a <code class="language-plaintext highlighter-rouge">poll_next</code> function on it. Well looking at the <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/buffered.rs.html#60">implementation of Stream for Buffered</a>, you can see a queue and a loop polling each item until they’re ready. <a href="https://docs.rs/futures-util/0.3.26/src/futures_util/stream/stream/buffer_unordered.rs.html#62"><code class="language-plaintext highlighter-rouge">BufferUnordered</code></a> is similar but doesn’t care about the order.</p>
<p>Right now we know how to get the futures module to poll out Futures, let’s put it all together.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="nf">.map</span><span class="p">(</span><span class="n">double_infallible</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="nf">.buffered</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="nf">.collect</span><span class="p">()</span>
<span class="err"> </span> <span class="err"> </span> <span class="k">.await</span><span class="p">;</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<p>Boom!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0, 2, 4, 6, 8]
</code></pre></div></div>
<h2 id="bring-back-the-result">Bring back the <code class="language-plaintext highlighter-rouge">Result</code></h2>
<p>Okay, now back at the beginning, with Iterators, I wanted a <code class="language-plaintext highlighter-rouge">Result<Vec<usize>, _></code>. As a reminder</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="n">Result</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span><span class="p">,</span> <span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span><span class="nf">.map</span><span class="p">(</span><span class="n">double_sync</span><span class="p">)</span><span class="nf">.collect</span><span class="p">();</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<p>So let’s stick <code class="language-plaintext highlighter-rouge">async</code> at the front of <code class="language-plaintext highlighter-rouge">double_sync</code> and rename it</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">async</span> <span class="k">fn</span> <span class="nf">double</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">usize</span><span class="p">)</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="nb">usize</span><span class="p">,</span> <span class="nb">String</span><span class="o">></span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span><span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="mi">3</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">Err</span><span class="p">(</span><span class="s">"I don't like three"</span><span class="nf">.to_owned</span><span class="p">())</span>
<span class="err"> </span> <span class="err"> </span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nf">Ok</span><span class="p">(</span><span class="n">x</span> <span class="o">*</span> <span class="mi">2</span><span class="p">)</span>
<span class="err"> </span> <span class="err"> </span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And plug this into the stream</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span>
<span class="nf">.map</span><span class="p">(</span><span class="n">double</span><span class="p">)</span>
<span class="nf">.buffered</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="nf">.collect</span><span class="p">()</span>
<span class="k">.await</span><span class="p">;</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Ok(0), Ok(2), Ok(4), Err("I don't like three"), Ok(8)]
</code></pre></div></div>
<p>We’ve seen that before. Let’s make the type <code class="language-plaintext highlighter-rouge">Result<Vec<_>, _></code> and then we can all go home</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="n">Result</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span><span class="p">,</span> <span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span>
<span class="nf">.map</span><span class="p">(</span><span class="n">double</span><span class="p">)</span>
<span class="nf">.buffered</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="nf">.collect</span><span class="p">()</span>
<span class="k">.await</span><span class="p">;</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<p>Go, <code class="language-plaintext highlighter-rouge">rustc</code>, go!</p>
<pre><code class="language-rust_errors">error[E0277]: the trait bound `Result<Vec<_>, _>: Default` is not satisfied
--> src/main.rs:75:9
|
75 | .await;
| ^^^^^^ the trait `Default` is not implemented for `Result<Vec<_>, _>`
|
= note: required for `Collect<Buffered<futures::stream::Map<futures::stream::Iter<std::ops::Range<usize>>, ...>>, ...>` to implement `futures::Future`
</code></pre>
<p>And another <strong>63</strong> lines of errors. Where’s that table, I’m going to flip it again 😡. Okay. Breathe. Look at the docs. Hey, what’s that in the <a href="https://docs.rs/futures/latest/futures/stream/index.html">futures::stream</a> module</p>
<h2 id="trystreamext"><a href="https://docs.rs/futures/latest/futures/stream/trait.TryStreamExt.html">TryStreamExt</a></h2>
<blockquote>
<p>Adapters specific to <code class="language-plaintext highlighter-rouge">Result</code>-returning streams</p>
</blockquote>
<p>Oh you sound ideal. And you have a <a href="https://docs.rs/futures/latest/futures/stream/trait.TryStreamExt.html#method.try_collect"><code class="language-plaintext highlighter-rouge">try_collect</code></a> which</p>
<blockquote>
<p>This combinator will collect all successful results of this stream
If an error happens then … the error will be returned.</p>
</blockquote>
<p>Let’s “try” it (and don’t forget to <code class="language-plaintext highlighter-rouge">use futures::TryStreamExt;</code>)</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">results</span><span class="p">:</span> <span class="n">Result</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="mi">_</span><span class="o">></span><span class="p">,</span> <span class="mi">_</span><span class="o">></span> <span class="o">=</span> <span class="nn">futures</span><span class="p">::</span><span class="nn">stream</span><span class="p">::</span><span class="nf">iter</span><span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">)</span>
<span class="nf">.map</span><span class="p">(</span><span class="n">double</span><span class="p">)</span>
<span class="nf">.buffered</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="nf">.try_collect</span><span class="p">()</span>
<span class="k">.await</span><span class="p">;</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{results:?}"</span><span class="p">);</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Err("I don't like three")
</code></pre></div></div>
<p>OH YEAH!!!!
Change the range to <code class="language-plaintext highlighter-rouge">4..10</code> and we get</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Ok([8, 10, 12, 14, 16, 18])
</code></pre></div></div>
<p>Right, I’m done, I’m going to bed. I hope this is both helpful and correct. Please let me know if it isn’t.</p>
Regex madness2016-06-16T00:00:00+00:00https://oxo42.github.io/regex/2016/06/16/regex-madness
<p>In order to make my brain melt, I just need to look at this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>((?:[^ ]|[^ ] [^ ])+)(?: +|$)
</code></pre></div></div>
<p>Q: What the hell does it do?
A: It parses a row where columns are separated by two or more spaces!
Q: Why the hell would you do that?
A: Because Huawei</p>
<p>Here’s my header line</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Route name Route Number Priority
</code></pre></div></div>
<p>and I want to get the field headers <code class="language-plaintext highlighter-rouge">Route name</code>, <code class="language-plaintext highlighter-rouge">Route Number</code> and <code class="language-plaintext highlighter-rouge">Priority</code> out of the above. Note that there are two spaces between <code class="language-plaintext highlighter-rouge">Route Number</code> and <code class="language-plaintext highlighter-rouge">Priority</code>.</p>
<p>Let’s simplify the problem to the point of uselessness and build up from there:</p>
<p>You have a single column containing multiple instances of a</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aaaaa
</code></pre></div></div>
<p>the regex would be</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(a+)$
</code></pre></div></div>
<p>The column can include a space, but only in the middle</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aaaaa
aa aa a
</code></pre></div></div>
<p>Here I want to match either an <code class="language-plaintext highlighter-rouge">a</code> or an <code class="language-plaintext highlighter-rouge">a a</code> one or more times. Let’s do this</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">a</code> or <code class="language-plaintext highlighter-rouge">a a</code> = <code class="language-plaintext highlighter-rouge">(a|a a)</code></li>
<li>That multiple times = <code class="language-plaintext highlighter-rouge">(a|a a)+</code></li>
<li>Capture the above by wrapping it with <code class="language-plaintext highlighter-rouge">()</code></li>
</ul>
<p>so</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>((a|a a)+)$
</code></pre></div></div>
<p>But look at that in <a href="https://regex101.com/r/oS3uZ1/1">regex101.com</a>. It leaves us with the problem that there are now two capturing groups. The inside parens are there only to provide the choice, we don’t really care abou them. Enter <a href="http://www.regular-expressions.info/brackets.html#noncap">Regular Expressions Non-Capturing Groups</a>. Put a <code class="language-plaintext highlighter-rouge">?:</code> at the beginning of the group:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>((?:a|a a)+)$
</code></pre></div></div>
<p>Nice! What next. Let’s have multiple columns separated by two spaces. What we’re actually looking for after our capturing is actually either two spaces <strong>OR</strong> the end of line <code class="language-plaintext highlighter-rouge">$</code>. So <code class="language-plaintext highlighter-rouge">( |$)</code>…. buuuut that creates another <a href="https://regex101.com/r/oS3uZ1/2">captured group</a>, therefore <code class="language-plaintext highlighter-rouge">(?: |$)</code>.</p>
<p><a href="https://regex101.com/r/oS3uZ1/3">Version 3</a> is</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>((?:a|a a)+)(?: |$)
</code></pre></div></div>
<p>Let’s change the separator to 2 or more spaces ` +`</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>((?:a|a a)+)(?: +|$)
</code></pre></div></div>
<p>And we’re nearly there. The only problem is the character <code class="language-plaintext highlighter-rouge">a</code> is quite limiting. What about if we were to change it to the opposite of a space <code class="language-plaintext highlighter-rouge">[^ ]</code>. I’ve saved this step for last because it visually complicates things. Here is the <a href="https://regex101.com/r/oS3uZ1/4">final regex</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>((?:[^ ]|[^ ] [^ ])+)(?: +|$)
</code></pre></div></div>
Splunk: Wrap a webhook for delivery to an HTTP Event collector2016-01-21T00:00:00+00:00https://oxo42.github.io/splunk/2016/01/21/splunk-wrap-a-webhook-for-delivery-to-an-http-event-collector
<p>It all started with <a href="http://help.fogcreek.com/10800/webhooks">FogBugz WebHooks</a>. I wanted to bring them into <a href="http://splunk.com">Splunk</a> and using the <a href="http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector">Splunk HTTP Event Collector</a> seems like a good fit.</p>
<p>There were a few problems with sending the raw data to a Splunk HTTP event collector:</p>
<ol>
<li>There was no way of configuring the Authorization header in the FogBugz webhook</li>
<li>The Splunk event collector expects the event data to be in an <code class="language-plaintext highlighter-rouge">event</code> field</li>
</ol>
<h3 id="fogbugz-webhook">FogBugz WebHook</h3>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="nl">"eventtype"</span><span class="p">:</span><span class="s2">"CaseEdited"</span><span class="p">,</span><span class="w"> </span><span class="nl">"casenumber"</span><span class="p">:</span><span class="s2">"123"</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h3 id="required-splunk-http-event-collector-event">Required Splunk HTTP Event Collector Event</h3>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="nl">"event"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nl">"eventtype"</span><span class="p">:</span><span class="s2">"CaseEdited"</span><span class="p">,</span><span class="w"> </span><span class="nl">"casenumber"</span><span class="p">:</span><span class="s2">"123"</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>I already have Nginx running on a reverse proxy (with SSL) sitting in front of my search head. What I want to do is take the FogBugz webhook data, wrap it into a Splunk event, add the Auth header and post to the Splunk Forwarder running on the same box. Like this:</p>
<p><img src="/assets/FogBugzWebhookArchitecture.png" alt="Webhook Architecture" /></p>
<h2 id="1-splunk-forwarder">1. Splunk Forwarder</h2>
<p>First, setup the HTTP input. I have created an app that I push out to the Nginx server with my deployment server</p>
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[http]</span>
<span class="py">disabled</span> <span class="p">=</span> <span class="s">0</span>
<span class="nn">[http://fogbugz-webhook]</span>
<span class="py">disabled</span> <span class="p">=</span> <span class="s">0</span>
<span class="py">index</span> <span class="p">=</span> <span class="s">main</span>
<span class="py">indexes</span> <span class="p">=</span> <span class="s">main</span>
<span class="py">sourcetype</span> <span class="p">=</span> <span class="s">fogbugz</span>
<span class="py">token</span> <span class="p">=</span> <span class="s">SOME-GUID</span>
</code></pre></div></div>
<p>When you restart the forwarder, this will start the HTTP Event Collector listening on port 8088.</p>
<h2 id="2-nginx">2. Nginx</h2>
<p>Next up you need to modify the Nginx config file to add the location <code class="language-plaintext highlighter-rouge">/fogbugz-GUID</code>. I put a GUID in the location just to make it unguessable.</p>
<div class="language-nginx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">location</span> <span class="n">/fogbugz-GUID</span> <span class="p">{</span>
<span class="kn">proxy_pass</span> <span class="s">https://localhost:8088/services/collector</span><span class="p">;</span>
<span class="kn">proxy_read_timeout</span> <span class="mi">90</span><span class="p">;</span>
<span class="kn">proxy_connect_timeout</span> <span class="mi">90</span><span class="p">;</span>
<span class="kn">proxy_redirect</span> <span class="no">off</span><span class="p">;</span>
<span class="kn">proxy_set_header</span> <span class="s">Host</span> <span class="nv">$host</span><span class="p">;</span>
<span class="kn">proxy_set_header</span> <span class="s">X-Real-IP</span> <span class="nv">$remote_addr</span><span class="p">;</span>
<span class="kn">proxy_set_header</span> <span class="s">X-Forwarded-For</span> <span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span>
<span class="c1"># wrap the fogbugz webhook body for splunk</span>
<span class="kn">proxy_set_body</span> <span class="s">"</span><span class="p">{</span><span class="kn">\"event\":</span><span class="nv">$request_body</span><span class="err">}</span><span class="s">"</span><span class="p">;</span>
<span class="c1"># Add the Splunk token into the Authorization header</span>
<span class="kn">proxy_set_header</span> <span class="s">Authorization</span> <span class="s">"Splunk</span> <span class="s">SOME-GUID"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">service nginx reload</code> will enabled the location</li>
<li><a href="https://gist.github.com/oxo42/ae3c8a0d4c05749aed50">Nginx-Proxy.conf</a> is a gist of my anonymised nginx config file that is running in production</li>
</ul>
<h2 id="3-fogbugz">3. FogBugz</h2>
<p>Create the webhook in FogBugz to point to your shiny new location.</p>
<p><img src="/assets/FogBugzWebhookSetup.png" alt="FogBugz Webhook Setup" /></p>
<h3 id="notes">Notes</h3>
<ul>
<li>There was another issue of opening up the firewall and maintaining SSL certificates which was solved by my solution</li>
<li>I can’t get the event time to line up with <code class="language-plaintext highlighter-rouge">eventtime</code> in the webhook data. If anyone can help, please holler</li>
</ul>
VMWare Workstation: Start VMs on user login2015-10-05T00:00:00+00:00https://oxo42.github.io/windows/2015/10/05/start-vmware-workstation-on-login
<p>I want to start a certain list of VMWare Workstation virtual machines on machine boot. I created a Scheduled task that on logon runs:</p>
<figure class="highlight"><pre><code class="language-batch" data-lang="batch"><span class="s2">"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"</span> <span class="na">-NoLogo -NonInteractive -File </span><span class="s2">"C:\Users\Tools\Start-VMs.ps1"</span></code></pre></figure>
<p>And the contents of <code class="language-plaintext highlighter-rouge">Start-VMs.ps1</code> are:</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="nv">$vmrun</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'</span><span class="w">
</span><span class="nv">$vmlist</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">@(</span><span class="s1">'C:\Virtual Machines\Ubuntu\Ubuntu.vmx'</span><span class="p">,</span><span class="w"> </span><span class="s1">'I:\Virtual Machines\UbuntuServer\UbuntuServer.vmx'</span><span class="p">)</span><span class="w">
</span><span class="nv">$authd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Get-Service</span><span class="w"> </span><span class="nt">-Name</span><span class="w"> </span><span class="nx">VMAuthdService</span><span class="w">
</span><span class="kr">if</span><span class="p">(</span><span class="nv">$authd</span><span class="o">.</span><span class="nf">Status</span><span class="w"> </span><span class="o">-ne</span><span class="w"> </span><span class="s1">'Running'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nv">$authd</span><span class="o">.</span><span class="nf">Start</span><span class="p">()</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kr">foreach</span><span class="p">(</span><span class="nv">$vm</span><span class="w"> </span><span class="kr">in</span><span class="w"> </span><span class="nv">$vmlist</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="o">&</span><span class="w"> </span><span class="nv">$vmrun</span><span class="w"> </span><span class="n">start</span><span class="w"> </span><span class="s2">"</span><span class="nv">$vm</span><span class="s2">"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
Splunk my Garmin - Get the data in - Part 12015-05-18T00:00:00+00:00https://oxo42.github.io/splunk/2015/05/18/splunk-my-garmin-part1
<h2 id="splunk-my-garmin-series">Splunk my Garmin Series</h2>
<ul>
<li><a href="splunk-my-garmin-part1.md">Part 1: Get the data in</a></li>
<li>[Part 2: Make sense of TrackPoint]</li>
<li>[Part 3: Build the app]</li>
<li>[Part 4: Investigate the API]</li>
<li>[Part 5: ???]</li>
</ul>
<p>Ever since <a href="http://blogs.splunk.com/author/dgreenwood/">Dave</a> wrote about <a href="http://blogs.splunk.com/2015/03/22/downhill-splunking-part-1/">Downhill Splunking</a> I’ve wanted to suck my <a href="https://connect.garmin.com/modern/profile/John_Oxley_">Garmin Connect</a> activities into Splunk and explore. This is the first of a multi-part post on my experience doing so.</p>
<h2 id="where-my-files-at">Where my files at?</h2>
<p>I’ve been using <a href="https://tapiriik.com/">Tapiriik</a> for a long time to sync my Garmin activities to Endomondo. At the time I decided what’s the harm in putting them in <a href="https://db.tt/a9NHe6F">Dropbox</a>. Turns out that was a great idea, because I now have a year’s worth of data to play with.</p>
<h2 id="splunk-it-up">Splunk it up!</h2>
<p>Start by creating a <code class="language-plaintext highlighter-rouge">fitness</code> app. All sourcetypes, macros, extractions, dashboards etc are going to go in here. Once I’ve got something useful, I’ll push it up to GitHub and publish on Splunkbase.</p>
<ul>
<li>Create index in splunk <code class="language-plaintext highlighter-rouge">fitness</code></li>
</ul>
<p>Put a monitor on the directory so files automatically get indexed.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd C:\users\Me\DropBox\Apps\tapiriik
splunk add monitor . -index fitness -sourcetype tcx
</code></pre></div></div>
<h2 id="what-is-tcx">What is TCX</h2>
<p>TCX is Training Center Database XML file. it looks like:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version='1.0' encoding='UTF-8'?></span>
<span class="nt"><TrainingCenterDatabase</span> <span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span> <span class="na">xmlns:ns2=</span><span class="s">"http://www.garmin.com/xmlschemas/UserProfile/v2"</span> <span class="na">xmlns:tpx=</span><span class="s">"http://www.garmin.com/xmlschemas/ActivityExtension/v2"</span> <span class="na">xmlns:ns5=</span><span class="s">"http://www.garmin.com/xmlschemas/ActivityGoals/v1"</span> <span class="na">xmlns:ns4=</span><span class="s">"http://www.garmin.com/xmlschemas/ProfileExtension/v1"</span> <span class="na">xmlns=</span><span class="s">"http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2"</span><span class="nt">></span>
<span class="nt"><Activities></span>
<span class="nt"><Activity</span> <span class="na">Sport=</span><span class="s">"Biking"</span><span class="nt">></span>
<span class="nt"><Id></span>2015-05-17T07:41:32.000Z<span class="nt"></Id></span>
<span class="nt"><Lap</span> <span class="na">StartTime=</span><span class="s">"2015-05-17T07:41:32.000Z"</span><span class="nt">></span>
<span class="nt"><TotalTimeSeconds></span>14599.468<span class="nt"></TotalTimeSeconds></span>
<span class="nt"><DistanceMeters></span>105244.0<span class="nt"></DistanceMeters></span>
<span class="nt"><MaximumSpeed></span>14.508333333333333<span class="nt"></MaximumSpeed></span>
<span class="nt"><Calories></span>2496<span class="nt"></Calories></span>
<span class="nt"><AverageHeartRateBpm><Value></span>134<span class="nt"></Value></AverageHeartRateBpm></span>
<span class="nt"><MaximumHeartRateBpm><Value></span>186<span class="nt"></Value></MaximumHeartRateBpm></span>
<span class="nt"><Intensity></span>Active<span class="nt"></Intensity></span>
<span class="nt"><Cadence></span>73<span class="nt"></Cadence></span>
<span class="nt"><TriggerMethod></span>Manual<span class="nt"></TriggerMethod></span>
<span class="nt"><Track></span>
<span class="c"><!-- Kajillions of TrackPoints --></span>
<span class="nt"></Track></span>
<span class="nt"><Extensions></span>
<span class="nt"><LX</span> <span class="na">xmlns=</span><span class="s">"http://www.garmin.com/xmlschemas/ActivityExtension/v2"</span><span class="nt">></span>
<span class="nt"><MaxBikeCadence></span>178<span class="nt"></MaxBikeCadence></span>
<span class="nt"></LX></span>
<span class="nt"></Extensions></span>
<span class="nt"></Lap></span>
<span class="nt"></Activity></span>
<span class="nt"></Activities></span>
<span class="nt"></TrainingCenterDatabase></span></code></pre></figure>
<p>There can be lots of laps per activity and lots of activities per file. The sourcetype that I define below will kind of cater to multiple activities.</p>
<h2 id="explore-the-data">Explore the data</h2>
<p>Now lets explore what we’ve got</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>index=fitness
</code></pre></div></div>
<p>Ooh that was a mistake. Splunk created an event for every single track point because it found multiple timestamps. Slow down and <code class="language-plaintext highlighter-rouge">index=fitness | delete</code> everything. Upload the file and create the sourcetype. We’ll move this to the app later.</p>
<h2 id="fix-up-the-sourcetype">Fix up the sourcetype</h2>
<p>Look at the <a href="http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Propsconf">props.conf</a> documentation for multiline events. There are a few things we need to set for this sourcetype</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">TIME_PREFIX = \<Id\></code> This seems to be the case on my Garmin activities. Not sure about other systems.</li>
<li><code class="language-plaintext highlighter-rouge">BREAK_ONLY_BEFORE = \<\?xml version</code> I’ve set this to the XML declaration header. Each file is a single event so don’t break. In the future, this will change to break for <code class="language-plaintext highlighter-rouge"><Activity></code>.</li>
<li><code class="language-plaintext highlighter-rouge">MAX_EVENTS = 999999999</code> Splunk breaks at 256 lines. I set this to 5 orders of magnitude higher than my biggest file</li>
<li><code class="language-plaintext highlighter-rouge">KV_MODE = xml</code> to get field extractions</li>
</ul>
<p>Save the sourcetype as <code class="language-plaintext highlighter-rouge">tcx</code> in the app <code class="language-plaintext highlighter-rouge">fitness</code>. I’m unsure about the TIME_PREFIX field, but we’re good to go for now. The props.conf stanza is</p>
<figure class="highlight"><pre><code class="language-ini" data-lang="ini"><span class="nn">[tcx]</span>
<span class="py">BREAK_ONLY_BEFORE</span> <span class="p">=</span> <span class="se">\<\?</span><span class="s">xml version</span>
<span class="py">MAX_EVENTS</span> <span class="p">=</span> <span class="s">999999999</span>
<span class="py">NO_BINARY_CHECK</span> <span class="p">=</span> <span class="s">true</span>
<span class="py">TIME_PREFIX</span> <span class="p">=</span> <span class="se">\<</span><span class="s">Id</span><span class="se">\></span>
<span class="py">category</span> <span class="p">=</span> <span class="s">Structured</span>
<span class="py">description</span> <span class="p">=</span> <span class="s">Garmin Training Center Database XML File</span>
<span class="py">disabled</span> <span class="p">=</span> <span class="s">false</span>
<span class="py">pulldown_type</span> <span class="p">=</span> <span class="s">true</span>
<span class="py">KV_MODE</span> <span class="p">=</span> <span class="s">xml</span></code></pre></figure>
<h2 id="data-upload---round-2">Data upload - round 2</h2>
<p>Now to re-index what’s in the directory. I didn’t feel like mucking about with resetting fishbucket etc so a quick powershell line</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="n">gci</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">splunk</span><span class="w"> </span><span class="nx">add</span><span class="w"> </span><span class="nx">oneshot</span><span class="w"> </span><span class="bp">$_</span><span class="w"> </span><span class="nt">-index</span><span class="w"> </span><span class="nx">fitness</span><span class="w"> </span><span class="nt">-sourcetype</span><span class="w"> </span><span class="nx">tcx</span><span class="w"> </span><span class="p">}</span></code></pre></figure>
<p>Let’s see what we’ve got</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sourcetype=tcx | timechart count by "TrainingCenterDatabase.Activities.Activity{@Sport}"
</code></pre></div></div>
<p>Wahey! We’re getting some data.</p>
<p>Looking at the actual TCX files, I’m not sure how I’m going to deal with multi-lap events. That’s a problem for later in the post. I want to get some data out of the lap, total time, distance, calories, etc.</p>
<h2 id="alias-some-fields">Alias some fields</h2>
<p>These are the fields that I have aliased in <a href="https://gist.github.com/oxo42/34d2d755027b109b986a">props.conf</a>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FIELDALIAS-Sport = "TrainingCenterDatabase.Activities.Activity{@Sport}" AS Sport
FIELDALIAS-Cadence = TrainingCenterDatabase.Activities.Activity.Lap.Cadence AS Cadence
</code></pre></div></div>
<p>Now for something I haven’t been able to discover previously. How has my cadence changed over time?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>index=fitness sourcetype=tcx Sport=Biking Cadence=* | timechart span=mon avg(Cadence)
</code></pre></div></div>
<p>This is a good start. I’ll look at pulling some information about the track points apart. I want to try to get parity with what garmin connect shows as statistics.</p>
Splunk: Get results for the previous business day2015-05-14T00:00:00+00:00https://oxo42.github.io/splunk/2015/05/14/previous-business-day<p>Save the following as a macro <code class="language-plaintext highlighter-rouge">previous_business_day</code> and export it globally</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[| stats count
| eval day_of_week=tonumber(strftime(now(), "%w"))
| eval earliest=case(day_of_week==0, "-2d@d", day_of_week==1, "-3d@d", 1==1, "-1d@d")
| eval latest=case(day_of_week==0, "-1d@d", day_of_week==1, "-2d@d", 1==1, "@d")
| return earliest, latest]
</code></pre></div></div>
<p>Then run your search</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sourcetype=foo `previous_business_day`
</code></pre></div></div>
Splunk: Dashboard Drilldowns2015-04-09T09:00:00+00:00https://oxo42.github.io/splunk/2015/04/09/splunk-drilldown
<p>This is a note to future-me on how to do row based drill downs on tables in a Splunk dashboard.</p>
<h1 id="set-a-token">Set a token</h1>
<p>Set the <code class="language-plaintext highlighter-rouge">drilldown</code> to <code class="language-plaintext highlighter-rouge">row</code> on the source table and set a <code class="language-plaintext highlighter-rouge">token</code> based on one of the clicked values, e.g. in a <code class="language-plaintext highlighter-rouge"><table></code>, you could do:</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><option</span> <span class="na">name=</span><span class="s">"drilldown"</span><span class="nt">></span>row<span class="nt"></option></span>
<span class="nt"><drilldown></span>
<span class="nt"><set</span> <span class="na">token=</span><span class="s">"travel_type"</span><span class="nt">></span>$row.type$<span class="nt"></set></span>
<span class="nt"></drilldown></span></code></pre></figure>
<p>The token can be used in further panels.</p>
<p>Setting the <code class="language-plaintext highlighter-rouge">depends</code> attribute of a panel downstream will hide it until the token has been populated</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><chart</span> <span class="na">depends=</span><span class="s">"$travel_type$"</span><span class="nt">></span></code></pre></figure>
<h1 id="open-a-link">Open a link</h1>
<p>Another type of drilldown is for a link (hint: link to a dashboard in your own instance with prepopulated tokens):</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><option</span> <span class="na">name=</span><span class="s">"drilldown"</span><span class="nt">></span>row<span class="nt"></option></span>
<span class="nt"><drilldown></span>
<span class="nt"><link</span> <span class="na">target=</span><span class="s">"_blank"</span><span class="nt">></span>/app/transaction_app/merchant_query?form.merchant_name=$row.merchant$<span class="nt"></link></span>
<span class="nt"></drilldown></span></code></pre></figure>
<p>When you link to your own splunk server, use the relative URL from <code class="language-plaintext highlighter-rouge">/app</code>. That way, you don’t need to worry about different user’s region settings or the server name changing.</p>
Windows: Add static routes to VPN connection automatically2015-03-25T13:53:00+00:00https://oxo42.github.io/windows/2015/03/25/windows-add-route-on-vpn-connect
<p>I have a company VPN connection that I do not want all my traffic to go over the link, only certain netblocks. In order to do this, you need to:</p>
<ol>
<li>Disable default gateway</li>
<li>Create netsh script to add the routes</li>
<li>
<p>Create a scheduled task to fire the netsh script when the link is connected</p>
</li>
<li>
<h1 id="disable-the-default-gateway">Disable the default gateway</h1>
<p>From the network & sharing center, open the properties for the VPN adapter</p>
</li>
</ol>
<p><img src="/assets/disable_default_gateway.png" alt="Disable default gateway" /></p>
<ol>
<li>
<h1 id="create-netsh-script-to-add-the-routes">Create netsh script to add the routes</h1>
</li>
</ol>
<p>Add in routes as you desire</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>interface ipv4
add route prefix=192.168.23.0/24 interface="My VPN" store=active
add route prefix=172.16.99.0/24 interface="My VPN" store=active
exit
</code></pre></div></div>
<ol>
<li>
<h1 id="create-a-scheduled-task-to-fire-on-link-up">Create a scheduled task to fire on link up</h1>
</li>
</ol>
<p>The following command will create the scheduled task (split onto multiple lines for readability)</p>
<div class="language-bat highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">schtasks</span> <span class="na">/create /F /TN </span><span class="s2">"VPN Connection Update"</span>
<span class="na">/TR </span><span class="s2">"netsh -f C:\path\to\VpnRoutes.netsh"</span>
<span class="na">/SC </span><span class="kd">ONEVENT</span> <span class="na">/EC </span><span class="kd">Application</span> <span class="na">/RL </span><span class="kd">HIGHEST</span>
<span class="na">/MO </span><span class="s2">"*[System[(Level=4 or Level=0) and (EventID=20225)]] and *[EventData[Data='My VPN']]"</span>
</code></pre></div></div>
<p><em>Warning</em> The scheduled task will not run when on battery and there is no command line setting for this. You’ll need to go into Task Scheduler and change this under the Conditions tab.</p>
<p>Another, and more flexible route would be to create a powershell script to run on connect and call it with</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Powershell.exe</span><span class="w"> </span><span class="nt">-WindowStyle</span><span class="w"> </span><span class="nx">Hidden</span><span class="w"> </span><span class="nt">-NonInteractive</span><span class="w"> </span><span class="nt">-NoProfile</span><span class="w"> </span><span class="nt">-Command</span><span class="w"> </span><span class="nx">C:\path\to\script.ps1</span><span class="w">
</span></code></pre></div></div>
Splunk: Add views to the navigation menu2015-03-24T15:00:00+00:00https://oxo42.github.io/splunk/2015/03/24/splunk-add-views-to-navigation
<p>In the app, open the navigation menu:</p>
<ul>
<li>Settings</li>
<li>User Interface</li>
<li>Navigation menus</li>
<li>Select the menu</li>
</ul>
<h1 id="views">Views</h1>
<ul>
<li>The <code class="language-plaintext highlighter-rouge">match</code> attribute is basically a string contains. It is <strong>NOT</strong> a regex :(</li>
<li><code class="language-plaintext highlighter-rouge">source="unclassified"</code> will take all views that have not yet been displayed.</li>
</ul>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><collection</span> <span class="na">label=</span><span class="s">"Transactions"</span><span class="nt">></span>
<span class="nt"><view</span> <span class="na">source=</span><span class="s">"unclassified"</span> <span class="na">match=</span><span class="s">"transaction"</span> <span class="nt">/></span>
<span class="nt"></collection></span>
<span class="nt"><collection</span> <span class="na">label=</span><span class="s">"Views"</span><span class="nt">></span>
<span class="nt"><view</span> <span class="na">source=</span><span class="s">"unclassified"</span> <span class="nt">/></span>
<span class="nt"></collection></span></code></pre></figure>
Splunk: Use a lookup to generate search terms in a subsearch2015-03-20T14:57:59+00:00https://oxo42.github.io/splunk/2015/03/20/splunk-multiple-lookups
<p>I am collecting SNMP data using my own <a href="https://github.com/oxo42/snmpmod">SNMP Modular Input Poller</a>. I’ve then got a number of graphs and such coming off it. The requirement is to build a table on a monthly basis of 95th percentile statistics for a selection of hosts and interface indexes. The challenge is we want to display a textual name not the host and interface. The output must look like:</p>
<table>
<thead>
<tr>
<th>host</th>
<th>ifIndex</th>
<th>name</th>
</tr>
</thead>
<tbody>
<tr>
<td>host1</td>
<td>1</td>
<td>Peer 1 Link A</td>
</tr>
<tr>
<td>host1</td>
<td>2</td>
<td>Peer 2 Link A</td>
</tr>
</tbody>
</table>
<p>Which I have saved to a <a href="http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/lookup">lookup</a> called <code class="language-plaintext highlighter-rouge">perc95_links</code>.</p>
<p>I want my search to be</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sourcetype=snmpip (host=host1 ifIndex=1) OR (host=host1 ifIndex=2) | `snmpif_traffic`
| stats perc95(mbpsIn) as "mbpsIn" perc95(mbpsOut) as "mbpsOut" by host ifIndex
| lookup perc95_links host ifIndex OUTPUT name
</code></pre></div></div>
<p><em>Note the lookup on two fields :)</em></p>
<p>Our friend to the rescue is <a href="http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/format">format</a>. By using the lookup as a generator</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>| inputlookup perc95_links | fields host ifIndex | format
</code></pre></div></div>
<p>we get the output</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>( (host="host1" AND ifIndex="1" ) OR ( host="host1" AND ifIndex="2" ) )
</code></pre></div></div>
<p>Then we use the lookup search as a subsearch and get:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sourcetype=snmpip [| inputlookup perc95_links | fields host ifIndex | format]] | `snmpif_traffic`
| stats perc95(mbpsIn) as "mbpsIn" perc95(mbpsOut) as "mbpsOut" by host ifIndex
| lookup perc95_links host ifIndex OUTPUT name
</code></pre></div></div>